<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Md Shamim on Medium]]></title>
        <description><![CDATA[Stories by Md Shamim on Medium]]></description>
        <link>https://medium.com/@shamimice03?source=rss-a96bc5a23088------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CHido3HpRibWVL_js2of_g.jpeg</url>
            <title>Stories by Md Shamim on Medium</title>
            <link>https://medium.com/@shamimice03?source=rss-a96bc5a23088------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:34:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shamimice03/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Secure Content Delivery with CloudFront Signed URLs and Cookies: A Complete Implementation Guide]]></title>
            <link>https://levelup.gitconnected.com/secure-content-delivery-with-cloudfront-signed-urls-and-cookies-a-complete-implementation-guide-7c0a03029d2d?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c0a03029d2d</guid>
            <category><![CDATA[aws-security]]></category>
            <category><![CDATA[aws-cloudfront]]></category>
            <category><![CDATA[aws-secrets-manager]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-s3]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Thu, 22 Jan 2026 19:48:58 GMT</pubDate>
            <atom:updated>2026-01-22T19:48:58.339Z</atom:updated>
            <content:encoded><![CDATA[<h4>Step-by-step implementation guide for protecting paid content using Amazon S3, CloudFront, and cryptographic access controls with Terraform and Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A3mpX6o0f7QiqMbyXi86XA.png" /><figcaption>Flow for cloudfront signed url and cookies</figcaption></figure><h4>Background:</h4><p>Let’s assume there are three categories of content: General, Premium, and VIP. A user’s subscription tier determines which categories they can access. General content is available to everyone, whereas Premium and VIP content are restricted and accessible only through a paid subscription.</p><p>The content is stored in Amazon S3 and delivered through Amazon CloudFront. To securely restrict access to paid content, CloudFront provides a robust mechanism using signed URLs and signed cookies. With this approach, only authorized users (Premium or VIP) receive time-limited, cryptographically signed access to the protected content.</p><p>In this article, we’ll walk through a step-by-step breakdown of how to implement this secure content delivery architecture using CloudFront signed URLs and Cookies.</p><p><strong>An example of S3 bucket folder structure:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rt85VaA4lBeePrh2mOOktw.png" /></figure><p><strong>After adding to the key groups:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmWEbfNzIosa-p7R9Zd5Mw.png" /></figure><h4>Solution Overview</h4><p>To implement this solution, we need to leverage several AWS services:</p><ul><li><strong>Amazon S3</strong> stores all content with strict access controls.</li><li><strong>Amazon CloudFront</strong> delivers content globally with low latency.</li><li><strong>CloudFront Signed URLs/Cookies</strong> provide cryptographic access verification.</li><li><strong>AWS Secrets Manager</strong> securely stores private keys and cloudfront public key id for signature generation.</li><li><strong>Origin Access Control (OAC)</strong> ensures S3 content is only accessible through CloudFront.</li></ul><blockquote><em>📦 Complete Project Repository: All code, configurations, and sample data used in this blog are available on GitHub:<br></em><a href="https://github.com/shamimice03/cloudfront-signed-url-and-cookies"><strong><em>https://github.com/shamimice03/cloudfront-signed-url-and-cookies</em></strong></a></blockquote><h4>Step1: Prepare S3 Bucket</h4><p>First, let’s create an S3 bucket to store our content. Here’s a Terraform configuration for a secure, private bucket →<strong> </strong><a href="https://github.com/shamimice03/cloudfront-signed-url-and-cookies/blob/main/terraform/s3.tf"><strong>Link</strong></a></p><pre>variable &quot;s3_bucket_name&quot; {<br>  description = &quot;Existing S3 bucket name&quot;<br>  type        = string<br>  default     = &quot;cloudterms-contents&quot;<br>}<br><br>module &quot;s3_bucket&quot; {<br>  source  = &quot;terraform-aws-modules/s3-bucket/aws&quot;<br>  version = &quot;5.10.0&quot;<br><br>  bucket = var.s3_bucket_name<br>  acl    = &quot;private&quot;<br><br>  control_object_ownership = true<br>  object_ownership         = &quot;ObjectWriter&quot;<br><br>  versioning = {<br>    enabled = false<br>  }<br>}</pre><p><strong>Important:</strong> Ensure all public access to the bucket is blocked. We’ll grant access exclusively through CloudFront using <strong>Origin Access Control</strong> in later steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6qLQYVh6Uwjdn0HNLi1Cg.png" /></figure><h4>Step2: Prepare Key Pair for Signature</h4><p>CloudFront signed URLs rely on asymmetric cryptography. The private key signs the URLs, while CloudFront uses the corresponding public key to verify authenticity.</p><p><strong>Generate a key pair using OpenSSL:</strong></p><pre># Create keys directory<br>mkdir -p terraform/keys<br><br># Generate private key (2048-bit RSA)<br>openssl genrsa -out terraform/keys/private_key.pem 2048<br><br># Generate public key<br>openssl rsa -pubout -in terraform/keys/private_key.pem -out terraform/keys/public_key.pem<br><br># Set proper permissions<br>chmod 600 terraform/keys/private_key.pem<br>chmod 644 terraform/keys/public_key.pem<br><br># View the keys<br>cat terraform/keys/public_key.pem<br>cat terraform/keys/private_key.pem</pre><p>After the keys are created, we need to upload the public key to CloudFront and organize it into a key group:</p><p><strong>Note: </strong>The key group allows us to manage multiple public keys and rotate them without disrupting our distribution configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uBuwOH9yvazpKIPcBHj_aQ.png" /></figure><p>Terraform code for that → <a href="https://github.com/shamimice03/cloudfront-signed-url-and-cookies/blob/main/terraform/secrets.tf"><strong>Link</strong></a></p><pre>variable &quot;environment&quot; {<br>  description = &quot;Environment name&quot;<br>  type        = string<br>  default     = &quot;poc&quot;<br>}<br><br># Register the public key with CloudFront<br>resource &quot;aws_cloudfront_public_key&quot; &quot;signing_key&quot; {<br>  name        = &quot;${var.environment}-signing-key&quot;<br>  encoded_key = file(&quot;${path.module}/keys/public_key.pem&quot;)<br>  comment     = &quot;Public key for CloudFront signed URLs - ${var.environment}&quot;<br>}<br><br># Create a key group for access management<br>resource &quot;aws_cloudfront_key_group&quot; &quot;premium_content&quot; {<br>  name    = &quot;${var.environment}-premium-key-group&quot;<br>  comment = &quot;Key group for premium content access&quot;<br><br>  items = [<br>    aws_cloudfront_public_key.signing_key.id<br>  ]<br>}</pre><p><strong>After uploading the public key:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eZnEHoQ5TqO5iZOUfMGUCQ.png" /></figure><p><strong>After adding the public key to the key group:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmWEbfNzIosa-p7R9Zd5Mw.png" /></figure><h4>Step 3: Storing Keys Securely In Secrets Manager</h4><p><strong>&gt; Cloudfront Public Key Id On Secrets Manager </strong>(Alternative: Parameter Store ):</p><p>We need to store the <strong>Cloudfront Public Key Id </strong>( shown above ) in a centralized location so that the application can reference it when generating signed URLs. This public key Id is must be used during URL generation, because it tells Cloudfront which public key to use when validating the incoming request signature.</p><p>When a user accesses the signed URL, Cloudfront uses the specified public key to verify the signature and determine whether the request is authorized.</p><pre># Store Key Pair ID<br>resource &quot;aws_secretsmanager_secret&quot; &quot;cloudfront_key_pair_id&quot; {<br>  name        = &quot;${var.environment}/cloudfront/key-pair-id&quot;<br>  description = &quot;CloudFront Key Pair ID&quot;<br>}<br><br>resource &quot;aws_secretsmanager_secret_version&quot; &quot;cloudfront_key_pair_id&quot; {<br>  secret_id     = aws_secretsmanager_secret.cloudfront_key_pair_id.id<br>  secret_string = aws_cloudfront_public_key.signing_key.id<br>}</pre><p><strong>&gt; Private Key On Secrets Manager:<br></strong>We can also store the private key in a centralized place so that it can be accessed by the application, which will be responsible for generating signed URLs and cookies using the private key.</p><p>To store the private key on Secrets Manager:</p><pre># Store CloudFront private key in Secrets Manager<br>resource &quot;aws_secretsmanager_secret&quot; &quot;cloudfront_private_key&quot; {<br>  name        = &quot;${var.environment}/cloudfront/private-key&quot;<br>  description = &quot;CloudFront private key for signed URLs&quot;<br>}<br><br>resource &quot;aws_secretsmanager_secret_version&quot; &quot;cloudfront_private_key&quot; {<br>  secret_id     = aws_secretsmanager_secret.cloudfront_private_key.id<br>  secret_string = file(&quot;${path.module}/keys/private_key.pem&quot;)<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LsKpHyHowVdSJSTnEmpkVw.png" /></figure><p><strong>Sample curl command to generate URL:</strong></p><pre>aws cloudfront sign \<br>  --url &quot;https://d3sor38qfc3ttx.cloudfront.net/premium/image1.jpg&quot; \<br>  --key-pair-id K1SD6LKTQ1RJJ7 \<br>  --private-key file://keys/private_key.pem \<br>  --date-less-than 1769018875</pre><p>More details → <a href="https://docs.aws.amazon.com/cli/latest/reference/cloudfront/sign.html"><strong>Link</strong></a></p><h4>Step 4: Configuring CloudFront Distribution</h4><p>Now let’s create our CloudFront distribution with different behaviors for each content tier:</p><p><strong>Key Point:</strong> The <strong>trusted_key_groups</strong> parameter is what enforces signature verification for premium and VIP paths.</p><p>Terraform code → <a href="https://github.com/shamimice03/cloudfront-signed-url-and-cookies/blob/main/terraform/cloudfront.tf"><strong>Link</strong></a></p><pre># Caching policy<br>data &quot;aws_cloudfront_cache_policy&quot; &quot;caching_optimized&quot; {<br>  name = &quot;Managed-CachingOptimized&quot;<br>}<br><br># CloudFront Distribution<br>resource &quot;aws_cloudfront_distribution&quot; &quot;content&quot; {<br>  enabled         = true<br>  is_ipv6_enabled = true<br>  comment         = &quot;${var.environment} - ${var.s3_bucket_name} Distribution&quot;<br>  price_class     = &quot;PriceClass_All&quot;<br><br>  # S3 Origin<br>  origin {<br>    domain_name              = module.s3_bucket.s3_bucket_bucket_regional_domain_name<br>    origin_id                = &quot;S3-${var.s3_bucket_name}&quot;<br>    origin_access_control_id = aws_cloudfront_origin_access_control.content.id<br>  }<br><br>  # Public content (no signature required)<br>  default_cache_behavior {<br>    target_origin_id       = &quot;S3-${var.s3_bucket_name}&quot;<br>    viewer_protocol_policy = &quot;redirect-to-https&quot;<br>    allowed_methods        = [&quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;]<br>    cached_methods         = [&quot;GET&quot;, &quot;HEAD&quot;]<br>    compress               = true<br><br>    cache_policy_id = data.aws_cloudfront_cache_policy.caching_optimized.id<br>  }<br><br>  # Premium content (requires signed URLs/cookies)<br>  ordered_cache_behavior {<br>    path_pattern           = &quot;/premium/*&quot;<br>    target_origin_id       = &quot;S3-${var.s3_bucket_name}&quot;<br>    viewer_protocol_policy = &quot;redirect-to-https&quot;<br>    allowed_methods        = [&quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;]<br>    cached_methods         = [&quot;GET&quot;, &quot;HEAD&quot;]<br>    compress               = true<br><br>    # IMPORTANT: Require signed URLs/cookies for this path<br>    trusted_key_groups = [aws_cloudfront_key_group.premium_content.id]<br><br>    cache_policy_id = data.aws_cloudfront_cache_policy.caching_optimized.id<br>  }<br><br>  # VIP content (requires signed URLs/cookies)<br>  ordered_cache_behavior {<br>    path_pattern           = &quot;/vip/*&quot;<br>    target_origin_id       = &quot;S3-${var.s3_bucket_name}&quot;<br>    viewer_protocol_policy = &quot;redirect-to-https&quot;<br>    allowed_methods        = [&quot;GET&quot;, &quot;HEAD&quot;, &quot;OPTIONS&quot;]<br>    cached_methods         = [&quot;GET&quot;, &quot;HEAD&quot;]<br>    compress               = true<br><br>    # IMPORTANT: Require signed URLs/cookies for this path<br>    trusted_key_groups = [aws_cloudfront_key_group.premium_content.id]<br><br>    cache_policy_id = data.aws_cloudfront_cache_policy.caching_optimized.id<br>  }<br><br>  # Geographic restrictions (none for POC)<br>  restrictions {<br>    geo_restriction {<br>      restriction_type = &quot;none&quot;<br>    }<br>  }<br><br>  # Use default CloudFront certificate<br>  viewer_certificate {<br>    cloudfront_default_certificate = true<br>  }<br><br>  tags = {<br>    Environment = var.environment<br>    Name        = &quot;${var.environment}-content-cdn&quot;<br>    Purpose     = &quot;POC&quot;<br>  }<br>}</pre><p><strong>Console View For Restricting Viewer Access:<br></strong>The key configuration here is associating the key group with the origin behavior. When a request arrives with a signed URL, CloudFront uses the public keys from this group to cryptographically validate the signature and confirm the request is legitimate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WvdvzdvNi4C-hi0LfxtiXw.png" /></figure><h4>Step 5: Configuring Origin Access Control</h4><p>Since the bucket is fully restricted, we need to explicitly grant access to the Cloudfront distribution we created. To do this, we need to create an Origin Access Control (OAC) and update the bucket policy accordingly.</p><p>Terraform code → <a href="https://github.com/shamimice03/cloudfront-signed-url-and-cookies/blob/main/terraform/oac.tf"><strong>Link</strong></a></p><pre># CloudFront Origin Access Control<br>resource &quot;aws_cloudfront_origin_access_control&quot; &quot;content&quot; {<br>  name                              = &quot;${var.environment}-${var.s3_bucket_name}-oac&quot;<br>  description                       = &quot;OAC for ${var.s3_bucket_name}&quot;<br>  origin_access_control_origin_type = &quot;s3&quot;<br>  signing_behavior                  = &quot;always&quot;<br>  signing_protocol                  = &quot;sigv4&quot;<br>}<br><br># S3 bucket policy - ONLY CloudFront can access<br>resource &quot;aws_s3_bucket_policy&quot; &quot;content&quot; {<br>  bucket = module.s3_bucket.s3_bucket_id<br><br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [<br>      {<br>        Sid    = &quot;AllowCloudFrontServicePrincipal&quot;<br>        Effect = &quot;Allow&quot;<br>        Principal = {<br>          Service = &quot;cloudfront.amazonaws.com&quot;<br>        }<br>        Action   = &quot;s3:GetObject&quot;<br>        Resource = &quot;${module.s3_bucket.s3_bucket_arn}/*&quot;<br>        Condition = {<br>          StringEquals = {<br>            &quot;AWS:SourceArn&quot; = aws_cloudfront_distribution.content.arn<br>          }<br>        }<br>      }<br>    ]<br>  })<br>}</pre><h4>Access Test</h4><p>Since general content is publicly accessible, we can test our setup by making requests to all three content tiers and observing how each responds based on its access configuration</p><pre># Test 1: General content (should be accessible - HTTP 200)<br>curl -I https://d3sor38qfc3ttx.cloudfront.net/general/general_image1.avif<br><br># Test 2: Premium content without signature (should be denied - HTTP 403)<br>curl -I https://d3sor38qfc3ttx.cloudfront.net/premium/image1.jpg<br><br># Test 3: VIP content without signature (should be denied - HTTP 403)<br>curl -I https://d3sor38qfc3ttx.cloudfront.net/vip/boat.png</pre><p>As expected, content other than the public (general) resources are blocked:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M6N1Ish00RQAXfz6Hpl9bA.png" /></figure><h4>Generate Signed URL</h4><p>With infrastructure in place, let’s generate signed URLs programmatically. Here’s a Python implementation → <a href="https://github.com/shamimice03/cloudfront-signed-url-and-cookies/blob/main/code/generate_signed_url.py"><strong>Link</strong></a></p><pre>import boto3<br>import datetime<br>from botocore.signers import CloudFrontSigner<br>from cryptography.hazmat.primitives import hashes, serialization<br>from cryptography.hazmat.primitives.asymmetric import padding<br><br>PRIVATE_KEY_SECRET_ID = &quot;poc/cloudfront/private-key&quot;<br>KEY_PAIR_ID_SECRET_ID = &quot;poc/cloudfront/key-pair-id&quot;<br>CLOUDFRONT_DOMAIN = &quot;d3sor38qfc3ttx.cloudfront.net&quot;<br>PATH = &quot;/premium/image1.jpg&quot;<br><br><br>def sign_url(https://medium.com/path,%20cf_domain,%20hours=1):<br>    try:<br>        # Get secrets from AWS Secrets Manager<br>        secrets_client = boto3.client(&quot;secretsmanager&quot;)<br>        private_key_pem = secrets_client.get_secret_value(<br>            SecretId=PRIVATE_KEY_SECRET_ID<br>        )[&quot;SecretString&quot;]<br>        key_pair_id = secrets_client.get_secret_value(SecretId=KEY_PAIR_ID_SECRET_ID)[<br>            &quot;SecretString&quot;<br>        ]<br><br>        # Load the private key for cryptography<br>        private_key = serialization.load_pem_private_key(<br>            private_key_pem.encode(), password=None<br>        )<br><br>        # Define the signing function that CloudFrontSigner will use<br>        def rsa_signer(message):<br>            # CloudFrontSigner expects a SHA1 signature for compatibility<br>            return private_key.sign(message, padding.PKCS1v15(), hashes.SHA1())<br><br>        # Create the CloudFront signer<br>        cf_signer = CloudFrontSigner(key_pair_id, rsa_signer)<br><br>        # Generate the signed URL<br>        expire_date = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(<br>            hours=hours<br>        )<br>        full_url = f&quot;https://{cf_domain}{path}&quot;<br><br>        signed_url = cf_signer.generate_presigned_url(https://medium.com/%3Cbr%3E%20%20%20%20%20%20%20%20%20%20%20%20full_url,%20date_less_than=expire_date%3Cbr%3E)<br><br>        return signed_url<br><br>    except Exception as e:<br>        print(f&quot;An error occurred: {e}&quot;)<br>        return None<br><br><br>if __name__ == &quot;__main__&quot;:<br>    # Generate a URL that is valid for the next 24 hours<br>    signed_url = sign_url(https://medium.com/PATH,%20CLOUDFRONT_DOMAIN,%20hours=24)<br><br>    if signed_url:<br>        print(&quot;\n--- Generated Signed URL ---&quot;)<br>        print(signed_url)<br>        print(&quot;---------------------------\n&quot;)<br><br>        # --- Generate and print the curl command ---<br>        print(&quot;--- Generated curl Command ---&quot;)<br>        print(f&#39;curl &quot;{signed_url}&quot; --output test.jpg&#39;)<br>        print(&quot;------------------------------\n&quot;)<br>    else:<br>        print(&quot;Failed to generate signed URL.&quot;)</pre><p><strong>The above code will generate the signed URL:</strong></p><pre>--- Generated Signed URL ---<br>https://d3sor38qfc3ttx.cloudfront.net/premium/image1.jpg?Expires=1769092187&amp;Signature=ETZ5xBlo6YNqCb98C~qPY85tlA2sQHbzTUGBbKBsoU3DEye54JT0y5sTcRs7Ggbgp0R0flkrse9xeGnxfRjdBQuGnx4pnxodNKiY5fQWkdlG3M0nOSlDg6T-tSSKZ0TJRIdO82pKGYbPcB8Mgw6nrq5UmWuBLxq36ioafioOlB9t-vVfey9LgRzWYy2cI5kLmfFBhQu5u9UclwkyVFHGLRQJT~bIBGt4PO~T0TwOGx~TCizbZZm9nqhIbI0bFV6JCIv1Xa9OoCfHxXCmVisejfQ5bW9nuff0HSGbFR2mtkWTPOHwwgltt~i3PzqK~2j9CU8ffHtD2qQVJSD4inl-fg__&amp;Key-Pair-Id=K1SD6LKTQ1RJJ7<br>---------------------------<br><br>--- Generated curl Command ---<br>curl &quot;https://d3sor38qfc3ttx.cloudfront.net/premium/image1.jpg?Expires=1769092187&amp;Signature=ETZ5xBlo6YNqCb98C~qPY85tlA2sQHbzTUGBbKBsoU3DEye54JT0y5sTcRs7Ggbgp0R0flkrse9xeGnxfRjdBQuGnx4pnxodNKiY5fQWkdlG3M0nOSlDg6T-tSSKZ0TJRIdO82pKGYbPcB8Mgw6nrq5UmWuBLxq36ioafioOlB9t-vVfey9LgRzWYy2cI5kLmfFBhQu5u9UclwkyVFHGLRQJT~bIBGt4PO~T0TwOGx~TCizbZZm9nqhIbI0bFV6JCIv1Xa9OoCfHxXCmVisejfQ5bW9nuff0HSGbFR2mtkWTPOHwwgltt~i3PzqK~2j9CU8ffHtD2qQVJSD4inl-fg__&amp;Key-Pair-Id=K1SD6LKTQ1RJJ7&quot; --output test.jpg<br>------------------------------</pre><h4><strong>Limitations of Signed URLs:</strong></h4><p>While this works perfectly for <strong>image1.jpg</strong>, attempting to access <strong>image2.jpg</strong> without generating a new signed URL results in an access denial:</p><pre>curl -v https://d3sor38qfc3ttx.cloudfront.net/premium/image2.jpg\?Expires\=1769092187\&amp;Signature\=ETZ5xBlo6YNqCb98C\~qPY85tlA2sQHbzTUGBbKBsoU3DEye54JT0y5sTcRs7Ggbgp0R0flkrse9xeGnxfRjdBQuGnx4pnxodNKiY5fQWkdlG3M0nOSlDg6T-tSSKZ0TJRIdO82pKGYbPcB8Mgw6nrq5UmWuBLxq36ioafioOlB9t-vVfey9LgRzWYy2cI5kLmfFBhQu5u9UclwkyVFHGLRQJT\~bIBGt4PO\~T0TwOGx\~TCizbZZm9nqhIbI0bFV6JCIv1Xa9OoCfHxXCmVisejfQ5bW9nuff0HSGbFR2mtkWTPOHwwgltt\~i3PzqK\~2j9CU8ffHtD2qQVJSD4inl-fg__\&amp;Key-Pair-Id\=K1SD6LKTQ1RJJ7</pre><pre>&gt; <br>* Request completely sent off<br>&lt; HTTP/2 403 <br>&lt; server: CloudFront<br>&lt; date: Wed, 21 Jan 2026 14:34:59 GMT<br>&lt; content-type: text/xml<br>&lt; content-length: 110<br>&lt; x-cache: Error from cloudfront<br>&lt; via: 1.1 1e897e027cbcf8ae91e6f0deddedab10.cloudfront.net (CloudFront)<br>&lt; x-amz-cf-pop: NRT12-P7<br>&lt; x-amz-cf-id: S8G7E0gNi9w8ybQ-3T5FiF44hUHz6fqTVfSNywDjsFalUQajuNMvPA==<br>&lt; <br>* Connection #0 to host d3sor38qfc3ttx.cloudfront.net left intact<br>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;&lt;Error&gt;&lt;Code&gt;AccessDenied&lt;/Code&gt;&lt;Message&gt;Access denied&lt;/Message&gt;&lt;/Error&gt;%        </pre><p>Each resource requires its own signed URL, which can become cumbersome when users need access to multiple files.</p><h4>Using Signed Cookies for Directory Access:</h4><p>Signed cookies solve this limitation by granting access to entire path patterns rather than individual files. Here’s how to generate them → <a href="https://github.com/shamimice03/cloudfront-signed-url-and-cookies/blob/main/code/generate_signed_cookie.py"><strong>Link</strong></a></p><pre>import boto3<br>import datetime<br>import json<br>import base64<br>from urllib.parse import quote<br>from cryptography.hazmat.primitives import hashes, serialization<br>from cryptography.hazmat.primitives.asymmetric import padding<br><br>PRIVATE_KEY_SECRET_ID = &quot;poc/cloudfront/private-key&quot;<br>KEY_PAIR_ID_SECRET_ID = &quot;poc/cloudfront/key-pair-id&quot;<br>CLOUDFRONT_DOMAIN = &quot;d3sor38qfc3ttx.cloudfront.net&quot;<br>PATH = &quot;/premium/*&quot;<br><br><br>def sign_cookies(path, cf_domain, hours=1):<br>    &quot;&quot;&quot;<br>    Generates signed CloudFront cookies for a given path.<br>    &quot;&quot;&quot;<br>    try:<br>        # Get secrets from AWS Secrets Manager<br>        secrets_client = boto3.client(&quot;secretsmanager&quot;)<br>        private_key_pem = secrets_client.get_secret_value(<br>            SecretId=PRIVATE_KEY_SECRET_ID<br>        )[&quot;SecretString&quot;]<br>        key_pair_id = secrets_client.get_secret_value(SecretId=KEY_PAIR_ID_SECRET_ID)[<br>            &quot;SecretString&quot;<br>        ]<br><br>        # Load the private key<br>        private_key = serialization.load_pem_private_key(<br>            private_key_pem.encode(), password=None<br>        )<br><br>        # Define the expiration time<br>        expire_date = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(<br>            hours=hours<br>        )<br><br>        # Create the JSON policy statement<br>        # This policy grants access to all files under the specified path until the expiration time.<br>        policy = {<br>            &quot;Statement&quot;: [<br>                {<br>                    &quot;Resource&quot;: f&quot;https://{cf_domain}{path}&quot;,<br>                    &quot;Condition&quot;: {<br>                        &quot;DateLessThan&quot;: {&quot;AWS:EpochTime&quot;: int(expire_date.timestamp())}<br>                    },<br>                }<br>            ]<br>        }<br><br>        # Encode the policy for the cookie<br>        # The policy must be base64-encoded and URL-safe.<br>        json_policy = json.dumps(policy).strip()<br>        encoded_policy = base64.b64encode(json_policy.encode()).decode()<br>        url_encoded_policy = quote(encoded_policy, safe=&quot;&quot;)<br><br>        # Sign the policy<br>        # The signature is created by signing the raw (not base64-encoded) JSON policy.<br>        signature = private_key.sign(<br>            json_policy.encode(),<br>            padding.PKCS1v15(),<br>            hashes.SHA1(),  # CloudFront requires SHA1 for RSA signatures<br>        )<br><br>        # Encode the signature for the cookie<br>        encoded_signature = base64.b64encode(signature).decode()<br>        url_encoded_signature = quote(encoded_signature, safe=&quot;&quot;)<br><br>        # Construct the signed cookies dictionary<br>        signed_cookies = {<br>            &quot;CloudFront-Policy&quot;: url_encoded_policy,<br>            &quot;CloudFront-Signature&quot;: url_encoded_signature,<br>            &quot;CloudFront-Key-Pair-Id&quot;: key_pair_id,<br>        }<br><br>        return signed_cookies<br><br>    except Exception as e:<br>        print(f&quot;An error occurred: {e}&quot;)<br>        return None<br><br><br>if __name__ == &quot;__main__&quot;:<br>    # Generate cookies that are valid for the next 24 hours<br>    signed_cookies = sign_cookies(PATH, CLOUDFRONT_DOMAIN, hours=24)<br><br>    if signed_cookies:<br>        print(&quot;\n--- Generated Signed Cookies ---&quot;)<br>        for name, value in signed_cookies.items():<br>            print(f&quot;{name}: {value}&quot;)<br>        print(&quot;--------------------------------\n&quot;)<br><br>        # --- Generate the curl command ---<br>        print(&quot;--- Generated curl Command ---&quot;)<br>        cookie_string = &quot;; &quot;.join([f&quot;{k}={v}&quot; for k, v in signed_cookies.items()])<br>        test_url = f&quot;https://{CLOUDFRONT_DOMAIN}/premium/image1.jpg --output test1.jpg&quot;<br><br>        print(f&#39;curl -b &quot;{cookie_string}&quot; {test_url}&#39;)<br>        print(&quot;------------------------------\n&quot;)<br><br>    else:<br>        print(&quot;Failed to generate signed cookies.&quot;)</pre><p><strong>Output will be something like this:</strong></p><pre><br>--- Generated Signed Cookies ---<br>CloudFront-Policy: eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6ICJodHRwczovL2Qzc29yMzhxZmMzdHR4LmNsb3VkZnJvbnQubmV0L3ByZW1pdW0vKiIsICJDb25kaXRpb24iOiB7IkRhdGVMZXNzVGhhbiI6IHsiQVdTOkVwb2NoVGltZSI6IDE3NjkwOTI3OTN9fX1dfQ%3D%3D<br>CloudFront-Signature: PYeKiTzINHkQpC1X4jXJTydyBipFC4u%2BU4PzwjMr7DgVeUss%2BugaF26gdZEWBKUMN%2FUSSca6yv3F39A%2FrT4Un6GGY9kXaF18qHLWxTL726HggIxtZrzxdI2XaTY4w1gF2Hp0Ja5twnX%2B2qFFrLy%2BXa5w1dfqBoWJjXmU%2BYnuBszZoi2wysr%2BWj5rmujsyi03LoB3InZCsvkCe1chVBoXV85qgRydCspsePao76y4t2ZW6kavYT37WyGYHOW5kwKs7sZtrqyMiTjxBR7P5%2FJoP69mgszORxXgkEho9htStShd9SYF5sOpFIf%2FL9fFpxPQKQbAdhR0zaRgE2LQ078QOg%3D%3D<br>CloudFront-Key-Pair-Id: K1SD6LKTQ1RJJ7<br>--------------------------------<br><br>--- Generated curl Command ---<br>curl -b &quot;CloudFront-Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6ICJodHRwczovL2Qzc29yMzhxZmMzdHR4LmNsb3VkZnJvbnQubmV0L3ByZW1pdW0vKiIsICJDb25kaXRpb24iOiB7IkRhdGVMZXNzVGhhbiI6IHsiQVdTOkVwb2NoVGltZSI6IDE3NjkwOTI3OTN9fX1dfQ%3D%3D; CloudFront-Signature=PYeKiTzINHkQpC1X4jXJTydyBipFC4u%2BU4PzwjMr7DgVeUss%2BugaF26gdZEWBKUMN%2FUSSca6yv3F39A%2FrT4Un6GGY9kXaF18qHLWxTL726HggIxtZrzxdI2XaTY4w1gF2Hp0Ja5twnX%2B2qFFrLy%2BXa5w1dfqBoWJjXmU%2BYnuBszZoi2wysr%2BWj5rmujsyi03LoB3InZCsvkCe1chVBoXV85qgRydCspsePao76y4t2ZW6kavYT37WyGYHOW5kwKs7sZtrqyMiTjxBR7P5%2FJoP69mgszORxXgkEho9htStShd9SYF5sOpFIf%2FL9fFpxPQKQbAdhR0zaRgE2LQ078QOg%3D%3D; CloudFront-Key-Pair-Id=K1SD6LKTQ1RJJ7&quot; https://d3sor38qfc3ttx.cloudfront.net/premium/image1.jpg --output test1.jpg<br>------------------------------</pre><h4><strong>Signed URLs vs. Signed Cookies:</strong></h4><p><strong>&gt; Use Signed URLs when:</strong></p><ul><li>We need to allow access to a single object</li><li>Creating expiring download links</li><li>Temporarily exposing specific content</li><li>Each file requires a different expiry time</li></ul><p><strong>&gt; Use Signed Cookies when:</strong></p><ul><li>Access is required for many files under the same path</li><li>Delivering streaming content or learning modules</li><li>Managing access through user sessions</li><li>Minimizing repeated signing operations</li></ul><blockquote><em>If you found this article useful, please click on the Follow 👉 and Clap 👏 buttons to show your support.<br>Thank you ❤</em><br>🔔<em> Find me on</em><strong><em> </em></strong><a href="https://www.linkedin.com/in/shamimice03/"><strong><em>LinkedIn </em></strong></a><strong><em>| </em></strong><a href="http://github.com/shamimice03"><strong><em>GitHub </em></strong></a><strong><em>| </em></strong><a href="https://twitter.com/shamimice03"><strong><em>Twitter</em></strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c0a03029d2d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/secure-content-delivery-with-cloudfront-signed-urls-and-cookies-a-complete-implementation-guide-7c0a03029d2d">Secure Content Delivery with CloudFront Signed URLs and Cookies: A Complete Implementation Guide</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building An Automated Data Pipeline Using AWS Glue: A Complete Guide]]></title>
            <link>https://aws.plainenglish.io/building-an-automated-data-pipeline-using-aws-glue-a-complete-guide-08d8a8b3dc00?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/08d8a8b3dc00</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws-athena]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Tue, 20 Jan 2026 17:47:44 GMT</pubDate>
            <atom:updated>2026-01-22T15:28:50.671Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to Build an Automated Data Pipeline Using AWS Glue, Terraform, and Apache Spark</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjHkHcrAsk87nvHhcD_u_Q.png" /><figcaption>data pipeline with aws glue</figcaption></figure><p>Data pipelines are essential to modern data engineering, turning raw data into actionable insights. This guide shows how to build a production-ready AWS Glue pipeline that converts JSON events into efficient, queryable Parquet files.</p><h4>What We’re Building</h4><p>We’ll create an automated pipeline that:</p><ul><li>Ingests raw JSON event data from S3.</li><li>Handles inconsistent schema structures.</li><li>Transforms and optimizes the data.</li><li>Partitions output for efficient querying.</li><li>Runs on a daily schedule without manual intervention.</li></ul><blockquote><em>📦 Complete Project Repository: All code, configurations, and sample data used in this blog are available on GitHub: </em><a href="https://github.com/shamimice03/data-pipeline-aws-glue"><strong><em>https://github.com/shamimice03/data-pipeline-aws-glue</em></strong></a></blockquote><h4><strong>What is ETL?</strong></h4><p>ETL stands for <strong>Extract, Transform, and Load. <br></strong>This is one of the famous data integration process which pull raw data, transforms it, and then loads it into various places as per requirements.</p><p><strong>Extract:</strong> In this step, data is collected from various sources such as object storage, IoT devices, databases, and other systems. In our case, we’re extracting JSON files from S3.</p><p><strong>Transform: </strong>This is the most critical phase, where the extracted data is cleaned, reshaped, and standardized. For example, let’s say we want to change the data type of a field from INT to STRING, change the timestamp format, or remove duplicate data, and so on. This phase may also include enriching the data by adding new fields, filtering records, or applying business rules.</p><p><strong>Load: </strong>In this step,<strong> t</strong>he transformed data is loaded into destination systems such as a data lake, data warehouse, or other storage and analytics platforms.</p><h4><strong>What is Glue Crawler?</strong></h4><p>An AWS Glue Crawler scans data from various sources, such as object storage and databases. It reads sample records from the data, infers the structure, and automatically determines details like column names and data types. Based on this information, it creates or updates tables in the AWS Glue Data Catalog.</p><h4><strong>What is Glue Data Catalog?</strong></h4><p>It’s a centralized metadata repository for all the data across various sources, as mentioned above. It’s like an interface that stores information about data formats, schema, and sources. Analytics services like Athena, Glue ETL, Amazon EMR, etc uses this data catalog to perform queries.</p><p>In this article, we will perform an hands on experiment to understand AWS Glue Crawler to create a data catalog table and AWS Glue ETL jobs to transform that data, so that Athena can do queries on that data.</p><h4>The Problem: Inconsistent Event Data</h4><p>Real-world data is messy. In our scenario, we have event data with inconsistent structures. Some events use nested objects while others use flat fields: <br>Follow the link for the dummy data → <a href="https://github.com/shamimice03/data-pipeline-aws-glue/tree/main/data"><strong>Link</strong></a></p><pre>// Event Type 1: Nested structure<br>{<br>  &quot;event_id&quot;: &quot;evt-8a1b2c3d&quot;,<br>  &quot;user&quot;: {<br>    &quot;id&quot;: &quot;user_456&quot;,<br>    &quot;age&quot;: 35,<br>    &quot;location&quot;: &quot;New York&quot;<br>  },<br>  &quot;product_details&quot;: {<br>    &quot;product_id&quot;: &quot;PROD-9876&quot;,<br>    &quot;category&quot;: &quot;Electronics&quot;<br>  },<br>  &quot;transaction&quot;: {<br>    &quot;amount&quot;: 129.99,<br>    &quot;currency&quot;: &quot;USD&quot;,<br>    &quot;timestamp&quot;: &quot;2023-11-20T14:32:10Z&quot;<br>  },<br>  &quot;feedback&quot;: {<br>    &quot;rating&quot;: 5,<br>    &quot;review_text&quot;: &quot;Absolutely love it!&quot;<br>  }<br>}<br><br>// Event Type 2: Flat structure<br>{<br>  &quot;event_id&quot;: &quot;evt-9d8c7b6a&quot;,<br>  &quot;user_id&quot;: &quot;user_123&quot;,<br>  &quot;user_age&quot;: 28,<br>  &quot;location&quot;: &quot;California&quot;,<br>  &quot;product_id&quot;: &quot;PROD-5555&quot;,<br>  &quot;product_category&quot;: &quot;Apparel&quot;,<br>  &quot;purchase_amount&quot;: 45.50,<br>  &quot;currency&quot;: &quot;USD&quot;,<br>  &quot;timestamp&quot;: &quot;2023-11-20T15:01:55Z&quot;,<br>  &quot;review&quot;: {<br>    &quot;rating&quot;: 4,<br>    &quot;review_text&quot;: &quot;Good quality&quot;<br>  }<br>}</pre><h4><strong>Solution:</strong></h4><p>Our pipeline follows this workflow:</p><pre>Raw JSON Events  ( s3 )<br>    ↓<br>Raw Data Crawler (discovers schema)<br>    ↓<br>ETL Job (transforms &amp; standardizes)<br>    ↓<br>Structured Data Crawler (catalogs output)<br>    ↓<br>Partitioned Parquet Files (query-ready)</pre><p>The entire process is orchestrated by an AWS Glue Workflow that runs daily at 10 PM JST.</p><h4>Implementation:</h4><p>Following is the step-by-step guide for this implementation:</p><h4><strong>Step 1: Setting Up the Raw Data Crawler</strong></h4><p>First, we will create a Glue Crawler to understand our raw data structure. Here’s the Terraform configuration →<a href="https://github.com/shamimice03/data-pipeline-aws-glue/blob/main/terraform/raw_event_crawler.tf"><strong>Link</strong></a></p><pre># Catalog database<br>resource &quot;aws_glue_catalog_database&quot; &quot;data_db&quot; {<br>  name = &quot;events_db&quot;<br>}<br><br># Glue Crawler for Raw Events<br>resource &quot;aws_glue_crawler&quot; &quot;raw_data_crawler&quot; {<br>  database_name = aws_glue_catalog_database.data_db.name<br>  name          = &quot;events-raw-crawler&quot;<br>  role          = aws_iam_role.glue_crawler_role.arn<br><br>  s3_target {<br>    path = &quot;s3://cloudterms-events-data/events/&quot;<br>  }<br><br>  configuration = jsonencode({<br>    Version = 1.0<br>    Grouping = {<br>      TableGroupingPolicy = &quot;CombineCompatibleSchemas&quot;<br>    }<br>    CreatePartitionIndex = true<br>  })<br><br>  schema_change_policy {<br>    update_behavior = &quot;UPDATE_IN_DATABASE&quot;<br>    delete_behavior = &quot;DEPRECATE_IN_DATABASE&quot;<br>  }<br><br>  recrawl_policy {<br>    recrawl_behavior = &quot;CRAWL_EVERYTHING&quot;<br>  }<br>}</pre><p>If we apply and run it, it will create a crawler and a catalog database. Once the crawler runs, it will generate the table along with its schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oGpOC4Zdz07IUiwDtZuvNA.png" /></figure><p>In the console schema will look like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQTtjtRzmPtvyY4nx-lhXw.png" /></figure><p>Schema in JSON:</p><pre>[<br>  {<br>    &quot;Name&quot;: &quot;event_id&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;user&quot;,<br>    &quot;Type&quot;: &quot;struct&lt;id:string,age:int,location:string&gt;&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;product_details&quot;,<br>    &quot;Type&quot;: &quot;struct&lt;product_id:string,category:string&gt;&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;transaction&quot;,<br>    &quot;Type&quot;: &quot;struct&lt;amount:double,currency:string,timestamp:string&gt;&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;feedback&quot;,<br>    &quot;Type&quot;: &quot;struct&lt;rating:int,review_text:string&gt;&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;product_id&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;product_category&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;purchase_amount&quot;,<br>    &quot;Type&quot;: &quot;double&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;currency&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;timestamp&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;review&quot;,<br>    &quot;Type&quot;: &quot;struct&lt;rating:int,review_text:string&gt;&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;user_id&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;user_age&quot;,<br>    &quot;Type&quot;: &quot;int&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;location&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  }<br>]</pre><p>As we can already see, there are some inconsistencies; we have user field nested with id , age , and location , while the same attributes also appear separately as user_age , user_id and location</p><p>Additionally, in some log entries, the field namedfeedback whereas in others it appears asreview</p><p>In the next step, we will use an ETL job to resolve these inconsistencies.</p><h4>Step 2: Creating the ETL Job</h4><p>One of the easiest ways to generate ETL Job scripts is by using the visual editor provided by AWS. More on <a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-intro-tutorial.html"><strong>here</strong></a></p><p>Let’s proceed with the visual editor and try to generate the script for this Job:</p><p>There are a lot of sources from which we can get data as a source. But in our case, we will use <strong>Glue Data Catalog</strong> as we already created a database and table in the previous step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*js7uN2Th7P3pIWQ8hLAFlw.png" /></figure><p><strong>Transformation 1: Resolving Field Inconsistencies:<br></strong>We will use SQL with <strong>COALESCE</strong> to handle both nested and flat structures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5zyQLDB1Eh3PzNuwdzdBuA.png" /></figure><pre>SELECT <br>  event_id,<br>  COALESCE(user.id, user_id) as user_id,<br>  COALESCE(user.age, user_age) as user_age,<br>  COALESCE(user.location, location) as location,<br>  COALESCE(product_details.product_id, product_id) as product_id,<br>  COALESCE(product_details.category, product_category) as product_category,<br>  COALESCE(transaction.amount, purchase_amount) as purchase_amount,<br>  COALESCE(transaction.currency, currency) as currency,<br>  COALESCE(transaction.timestamp, timestamp) as timestamp,<br>  COALESCE(feedback.rating, review.rating) as rating,<br>  COALESCE(feedback.review_text, review.review_text) as review_text<br>FROM myDataSource</pre><p>This query tries the nested field first, falls back to the flat field if null, ensuring we capture data regardless of structure.</p><p><strong>Transformation 2: Data Type Optimization<br></strong>In this step, we will update the data types of the <strong>timestamp</strong> and <strong>rating</strong> fields. Previously, the <strong>timestamp</strong> was stored as a <strong>string</strong> and the <strong>rating</strong> as an <strong>integer</strong>. We will convert the <strong>timestamp</strong> to a proper <strong>timestamp</strong> data type and the <strong>rating</strong> to a <strong>double</strong>.</p><blockquote><em>timestamp: </em><strong><em>string → timestamp</em></strong><em><br>rating: </em><strong><em>int → double</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nz_WHAindEORAPOJNV_xUA.png" /></figure><p><strong>Transformation 3: Adding Partition Columns <br></strong>For efficient querying, we will use the following query to generate a partition key; it will add new fields to our existing schema. For year , month , day and hour</p><pre>SELECT<br>    *,<br>    year(timestamp) AS year,<br>    month(timestamp) AS month,<br>    dayofmonth(timestamp) AS day,<br>    hour(timestamp) AS hour<br>FROM myDataSource</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DuoEOz2t5iMJQAn_UWHh4Q.png" /></figure><p><strong>Transformation 4: Writing to S3<br></strong>Finally, we load the transformed data as compressed Parquet files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bRbjWK0_u8qtLtBf-oYCIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xmzg5Jb9E3JUGj3YH4omng.png" /></figure><p>Data will be stored in S3 like this</p><pre>/year=YYYY/<br>  /month=MM/<br>    /day=DD/<br>      /hour=HH/</pre><p>Final look at the visual editor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5aSLdjD4YuIrBtz0Bqg6Q.png" /></figure><p>Final script we got from the visual editor is →<a href="https://github.com/shamimice03/data-pipeline-aws-glue/blob/main/glue-assets/scripts/events-etl.py"><strong>Link</strong></a></p><pre>import sys<br>from awsglue.transforms import *<br>from awsglue.utils import getResolvedOptions<br>from pyspark.context import SparkContext<br>from awsglue.context import GlueContext<br>from awsglue.job import Job<br>from awsgluedq.transforms import EvaluateDataQuality<br>from awsglue import DynamicFrame<br><br><br>def sparkSqlQuery(glueContext, query, mapping, transformation_ctx) -&gt; DynamicFrame:<br>    for alias, frame in mapping.items():<br>        frame.toDF().createOrReplaceTempView(alias)<br>    result = spark.sql(query)<br>    return DynamicFrame.fromDF(result, glueContext, transformation_ctx)<br><br><br>args = getResolvedOptions(sys.argv, [&quot;JOB_NAME&quot;])<br>sc = SparkContext()<br>glueContext = GlueContext(sc)<br>spark = glueContext.spark_session<br>job = Job(glueContext)<br>job.init(args[&quot;JOB_NAME&quot;], args)<br><br># Default ruleset used by all target nodes with data quality enabled<br>DEFAULT_DATA_QUALITY_RULESET = &quot;&quot;&quot;<br>    Rules = [<br>        ColumnCount &gt; 0<br>    ]<br>&quot;&quot;&quot;<br><br># Script generated for node AWS Glue Data Catalog<br>AWSGlueDataCatalog_node1768740156292 = glueContext.create_dynamic_frame.from_catalog(<br>    database=&quot;events_db&quot;,<br>    table_name=&quot;events&quot;,<br>    transformation_ctx=&quot;AWSGlueDataCatalog_node1768740156292&quot;,<br>)<br><br># Script generated for node SQL Query<br>SqlQuery1 = &quot;&quot;&quot;<br>SELECT <br>  event_id,<br>  COALESCE(user.id, user_id) as user_id,<br>  COALESCE(user.age, user_age) as user_age,<br>  COALESCE(user.location, location) as location,<br>  COALESCE(product_details.product_id, product_id) as product_id,<br>  COALESCE(product_details.category, product_category) as product_category,<br>  COALESCE(transaction.amount, purchase_amount) as purchase_amount,<br>  COALESCE(transaction.currency, currency) as currency,<br>  COALESCE(transaction.timestamp, timestamp) as timestamp,<br>  COALESCE(feedback.rating, review.rating) as rating,<br>  COALESCE(feedback.review_text, review.review_text) as review_text<br>FROM myDataSource<br>&quot;&quot;&quot;<br>SQLQuery_node1768740177246 = sparkSqlQuery(<br>    glueContext,<br>    query=SqlQuery1,<br>    mapping={&quot;myDataSource&quot;: AWSGlueDataCatalog_node1768740156292},<br>    transformation_ctx=&quot;SQLQuery_node1768740177246&quot;,<br>)<br><br># Script generated for node Change Schema<br>ChangeSchema_node1768740734843 = ApplyMapping.apply(<br>    frame=SQLQuery_node1768740177246,<br>    mappings=[<br>        (&quot;event_id&quot;, &quot;string&quot;, &quot;event_id&quot;, &quot;string&quot;),<br>        (&quot;user_id&quot;, &quot;string&quot;, &quot;user_id&quot;, &quot;string&quot;),<br>        (&quot;user_age&quot;, &quot;int&quot;, &quot;user_age&quot;, &quot;int&quot;),<br>        (&quot;location&quot;, &quot;string&quot;, &quot;user_location&quot;, &quot;string&quot;),<br>        (&quot;product_id&quot;, &quot;string&quot;, &quot;product_id&quot;, &quot;string&quot;),<br>        (&quot;product_category&quot;, &quot;string&quot;, &quot;product_category&quot;, &quot;string&quot;),<br>        (&quot;purchase_amount&quot;, &quot;double&quot;, &quot;purchase_amount&quot;, &quot;double&quot;),<br>        (&quot;currency&quot;, &quot;string&quot;, &quot;currency&quot;, &quot;string&quot;),<br>        (&quot;timestamp&quot;, &quot;string&quot;, &quot;timestamp&quot;, &quot;timestamp&quot;),<br>        (&quot;rating&quot;, &quot;int&quot;, &quot;rating&quot;, &quot;double&quot;),<br>        (&quot;review_text&quot;, &quot;string&quot;, &quot;review_text&quot;, &quot;string&quot;),<br>    ],<br>    transformation_ctx=&quot;ChangeSchema_node1768740734843&quot;,<br>)<br><br># Script generated for node SQL Query<br>SqlQuery0 = &quot;&quot;&quot;<br>SELECT<br>    *,<br>    year(timestamp) AS year,<br>    month(timestamp) AS month,<br>    dayofmonth(timestamp) AS day,<br>    hour(timestamp) AS hour<br>FROM myDataSource<br>&quot;&quot;&quot;<br>SQLQuery_node1768740795144 = sparkSqlQuery(<br>    glueContext,<br>    query=SqlQuery0,<br>    mapping={&quot;myDataSource&quot;: ChangeSchema_node1768740734843},<br>    transformation_ctx=&quot;SQLQuery_node1768740795144&quot;,<br>)<br><br># Script generated for node Amazon S3<br>EvaluateDataQuality().process_rows(<br>    frame=SQLQuery_node1768740795144,<br>    ruleset=DEFAULT_DATA_QUALITY_RULESET,<br>    publishing_options={<br>        &quot;dataQualityEvaluationContext&quot;: &quot;EvaluateDataQuality_node1768740683804&quot;,<br>        &quot;enableDataQualityResultsPublishing&quot;: True,<br>    },<br>    additional_options={<br>        &quot;dataQualityResultsPublishing.strategy&quot;: &quot;BEST_EFFORT&quot;,<br>        &quot;observations.scope&quot;: &quot;ALL&quot;,<br>    },<br>)<br>AmazonS3_node1768741302064 = glueContext.write_dynamic_frame.from_options(<br>    frame=SQLQuery_node1768740795144,<br>    connection_type=&quot;s3&quot;,<br>    format=&quot;glueparquet&quot;,<br>    connection_options={<br>        &quot;path&quot;: &quot;s3://cloudterms-events-data-outputs&quot;,<br>        &quot;partitionKeys&quot;: [&quot;year&quot;, &quot;month&quot;, &quot;day&quot;, &quot;hour&quot;],<br>    },<br>    format_options={&quot;compression&quot;: &quot;gzip&quot;},<br>    transformation_ctx=&quot;AmazonS3_node1768741302064&quot;,<br>)<br><br>job.commit()</pre><h4>Step 3: Cataloging the Output</h4><p>Now we can create a new Crawler for the structured/transformed data → <a href="https://github.com/shamimice03/data-pipeline-aws-glue/blob/main/terraform/structured_event_crawler.tf"><strong>Link</strong></a></p><pre># Catalog database<br>resource &quot;aws_glue_catalog_database&quot; &quot;structured_data_db&quot; {<br>  name = &quot;structured_events_db&quot;<br>}<br><br># Glue Crawler<br>resource &quot;aws_glue_crawler&quot; &quot;structured_data_crawler&quot; {<br>  database_name = aws_glue_catalog_database.structured_data_db.name<br>  name          = &quot;structured-events-crawler&quot;<br>  role          = aws_iam_role.glue_crawler_role.arn<br><br>  s3_target {<br>    path = &quot;s3://${local.output_data_bucket}&quot;<br>  }<br><br>  # Crawl partitioned Parquet data<br>  configuration = jsonencode({<br>    Version = 1.0<br>    Grouping = {<br>      TableGroupingPolicy = &quot;CombineCompatibleSchemas&quot;<br>    }<br>    CreatePartitionIndex = true<br>  })<br><br><br>  schema_change_policy {<br>    update_behavior = &quot;UPDATE_IN_DATABASE&quot;<br>    delete_behavior = &quot;DEPRECATE_IN_DATABASE&quot;<br>  }<br><br>  recrawl_policy {<br>    recrawl_behavior = &quot;CRAWL_EVERYTHING&quot;<br>  }<br>}</pre><p>This crawler creates a clean, partitioned table ready for analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gIr134mdyMrZBiiETv1Uxw.png" /></figure><p>The transformed event data schema looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IF8bG5yWBkNn163KEm5IFw.png" /></figure><p>Schema in JSON:</p><pre>[<br>  {<br>    &quot;Name&quot;: &quot;event_id&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;user_id&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;user_age&quot;,<br>    &quot;Type&quot;: &quot;int&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;user_location&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;product_id&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;product_category&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;purchase_amount&quot;,<br>    &quot;Type&quot;: &quot;double&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;currency&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;timestamp&quot;,<br>    &quot;Type&quot;: &quot;timestamp&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;rating&quot;,<br>    &quot;Type&quot;: &quot;double&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;review_text&quot;,<br>    &quot;Type&quot;: &quot;string&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;year&quot;,<br>    &quot;Type&quot;: &quot;string&quot;,<br>    &quot;PartitionKey&quot;: &quot;Partition (0)&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;month&quot;,<br>    &quot;Type&quot;: &quot;string&quot;,<br>    &quot;PartitionKey&quot;: &quot;Partition (1)&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;day&quot;,<br>    &quot;Type&quot;: &quot;string&quot;,<br>    &quot;PartitionKey&quot;: &quot;Partition (2)&quot;<br>  },<br>  {<br>    &quot;Name&quot;: &quot;hour&quot;,<br>    &quot;Type&quot;: &quot;string&quot;,<br>    &quot;PartitionKey&quot;: &quot;Partition (3)&quot;<br>  }<br>]</pre><h4>Step 4: Orchestrating with Workflows</h4><p>AWS Glue Workflows tie everything together, which will allow us to trigger a chain operation like:</p><p><strong>Raw Crawler → ETL Job → Structured Events Crawler</strong></p><pre># -----------------------------------------------------------------------------<br># WORKFLOW: Raw Crawler → ETL Job → Structured Events Crawler<br># -----------------------------------------------------------------------------<br><br># Glue Workflow to orchestrate the complete pipeline<br>resource &quot;aws_glue_workflow&quot; &quot;etl_pipeline&quot; {<br>  name = &quot;events-processing-pipeline&quot;<br>}<br><br># Trigger 0: Start the workflow on schedule (10 PM JST / 1 PM UTC daily)<br>resource &quot;aws_glue_trigger&quot; &quot;start_workflow&quot; {<br>  name          = &quot;start-workflow&quot;<br>  workflow_name = aws_glue_workflow.etl_pipeline.name<br>  type          = &quot;SCHEDULED&quot;<br>  schedule      = &quot;cron(0 13 * * ? *)&quot; # 1 PM UTC is 10 PM JST (UTC+9)<br><br>  actions {<br>    crawler_name = &quot;events-raw-crawler&quot;<br>  }<br>}<br><br># Trigger 1: Start ETL job when raw crawler succeeds<br>resource &quot;aws_glue_trigger&quot; &quot;run_etl_after_raw_crawl&quot; {<br>  name          = &quot;run-etl-after-raw-crawl&quot;<br>  workflow_name = aws_glue_workflow.etl_pipeline.name<br>  type          = &quot;CONDITIONAL&quot;<br><br>  predicate {<br>    conditions {<br>      crawler_name = &quot;events-raw-crawler&quot;<br>      crawl_state  = &quot;SUCCEEDED&quot;<br>    }<br>    logical = &quot;AND&quot;<br>  }<br><br>  actions {<br>    job_name = aws_glue_job.test_1_etl.name<br>  }<br>}<br><br># Trigger 2: Run structured events crawler after ETL job succeeds<br>resource &quot;aws_glue_trigger&quot; &quot;run_structured_crawler_after_etl&quot; {<br>  name          = &quot;run-structured-crawler-after-etl&quot;<br>  workflow_name = aws_glue_workflow.etl_pipeline.name<br>  type          = &quot;CONDITIONAL&quot;<br><br>  predicate {<br>    conditions {<br>      job_name = aws_glue_job.test_1_etl.name<br>      state    = &quot;SUCCEEDED&quot;<br>    }<br>    logical = &quot;AND&quot;<br>  }<br><br>  actions {<br>    crawler_name = &quot;structured-events-crawler&quot;<br>  }<br>}<br></pre><p>Workflow on AWS Console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oFvdLm6ZT1X1QfJrDl09fA.png" /></figure><h4>Step5: Querying the Data</h4><p>Once the pipeline completes, you can query your data using Amazon Athena:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BBiKo7e9TrsxM2WdXzlwPQ.png" /></figure><p><strong>Sample Query:</strong></p><pre>SELECT <br>    user_location,<br>    product_category,<br>    COUNT(*) as purchase_count,<br>    AVG(purchase_amount) as avg_amount,<br>    AVG(rating) as avg_rating<br>FROM structured_events_db.cloudterms_events_data_outputs<br>WHERE year = &#39;2023&#39; <br>  AND month = &#39;11&#39;<br>  AND product_category = &#39;Electronics&#39;<br>GROUP BY user_location, product_category<br>ORDER BY purchase_count DESC;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Rka64-AltcZv7yFb5xL1g.png" /></figure><blockquote><em>If you found this article useful, please click on the Follow 👉 and Clap 👏 buttons to show your support</em><strong><em>.</em></strong><em><br>Thank you ❤</em><br>🔔<em> Find me on</em><strong><em> </em></strong><a href="https://www.linkedin.com/in/shamimice03/"><strong><em>LinkedIn </em></strong></a><strong><em>| </em></strong><a href="http://github.com/shamimice03"><strong><em>GitHub </em></strong></a><strong><em>| </em></strong><a href="https://twitter.com/shamimice03"><strong><em>Twitter</em></strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08d8a8b3dc00" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/building-an-automated-data-pipeline-using-aws-glue-a-complete-guide-08d8a8b3dc00">Building An Automated Data Pipeline Using AWS Glue: A Complete Guide</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Security Series: AWS Access Key is Compromised. Now What? An Incident Response Playbook.]]></title>
            <link>https://medium.com/kubehub/aws-security-series-aws-access-key-is-compromised-now-what-an-incident-response-playbook-c568327fc7f6?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/c568327fc7f6</guid>
            <category><![CDATA[access-control]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[aws-iam]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-security]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 17:18:30 GMT</pubDate>
            <atom:updated>2026-01-19T17:24:40.781Z</atom:updated>
            <content:encoded><![CDATA[<h4>A real-world playbook for responding to exposed AWS access keys without breaking production</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*WfuZiYRbKUTQuSjn.png" /></figure><p>In the world of AWS, it’s always best practice to <strong>use IAM roles whenever possible.</strong> IAM roles provide temporary, automatically rotated credentials that are the gold standard for security. But reality is not the same. What happens when we need to integrate with a third-party service that only supports static, long-term credentials? This is where the IAM user with an access key becomes a necessary…</p><p>So, the critical moment arrives: <strong>an access key is exposed. What to do?</strong></p><h4>Phase 1: Assess the situation</h4><p>Our first instinct might be to hit the delete button-and fast. But hold on. A rash deletion could instantly break our production environment. A smarter, safer first move is to <strong>deactivate</strong> the key. Think of it as putting the key on ‘pause.’ It immediately stops working, giving you a crucial window to assess the impact on the applications.</p><p>Often, AWS detects this kind of exposure. It might send a notification and even proactively attach the <a href="https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSCompromisedKeyQuarantineV3.html">AWSCompromisedKeyQuarantineV3</a> policy to the user to help limit the damage.</p><h4>Phase 2: The Hidden Threat</h4><p>We’ve deactivated the key, created a new set, and updated our applications. But then, when we check CloudTrail, and shocked!!! Unauthorized API calls are still happening.</p><p><strong>How is this possible?<br></strong>The attacker was one step ahead. Before we deactivated the main key, they used it to generate <strong>temporary session tokens</strong> using a command like this:</p><pre>aws sts get-session-token --duration-seconds 129600 # (36 hours)</pre><p>These tokens are independent of the original key. They can remain valid for up to 36 hours, allowing the attacker to maintain a persistent backdoor even after the original key is compromised.</p><h4>Phase 3: Add Deny Policy to restrict Temporary Credentials</h4><p>We can create a “time-based fence” with an IAM policy to block <em>new</em> temporary tokens; Therefore, we can use a policy like the following:</p><pre>{ &quot;Version&quot;: &quot;2012-10-17&quot;, &quot;Statement&quot;: [ { &quot;Sid&quot;: &quot;DenyAccessWithTemporaryCredentialsIssuedAfterCompromise&quot;, &quot;Effect&quot;: &quot;Deny&quot;, &quot;Action&quot;: &quot;*&quot;, &quot;Resource&quot;: &quot;*&quot;, &quot;Condition&quot;: { &quot;DateLessThan&quot;: { &quot;aws:TokenIssueTime&quot;: &quot;2025-12-08T16:00:00Z&quot; } } } ] }</pre><p>Deny all temporary credentials that were created before the exposure of access keys.</p><h4>Phase 4: The Recovery — Restoring Access Safely</h4><p>It’s time to rebuild. The goal is to restore access without re-introducing the same risk.</p><ol><li><strong>Create New Credentials:</strong> The cleanest approach is to create brand new access keys. And review the policy and apply the <strong>principle of least privilege</strong>, granting only the permissions necessary for its job.</li><li><strong>Update All Systems:</strong> Update every application, script, configuration file, and secret store (like AWS Secrets Manager) with the new credentials.</li></ol><h4>Phase 5: The Post-Mortem — Hardening Our Defenses</h4><p>The incident is over, but the work isn’t. Now it’s time to learn from the attack and strengthen our defenses.</p><ol><li><strong>Forensic Analysis with CloudTrail:</strong> Dive into CloudTrail logs. Filter events by the compromised user’s name to see every API call they made. What data did they access? Did they try to create new users, roles, or backdoors? This log is our forensic evidence of the attack’s scope.</li><li><strong>Threat Detection with GuardDuty:</strong> We can use Amazon GuardDuty is an intelligent threat detection service that was likely monitoring our account during the attack. Review its findings for indicators of compromise, such as unusual API behavior or attempts to escalate privileges. GuardDuty can help us uncover other malicious activity we might have missed.</li></ol><p><em>Originally published at </em><a href="https://dev.to/shamimice03/aws-security-series-aws-access-key-is-compromised-now-what-an-incident-response-playbook-448b"><em>https://dev.to</em></a><em> on December 8, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c568327fc7f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kubehub/aws-security-series-aws-access-key-is-compromised-now-what-an-incident-response-playbook-c568327fc7f6">AWS Security Series: AWS Access Key is Compromised. Now What? An Incident Response Playbook.</a> was originally published in <a href="https://medium.com/kubehub">blog.cloudterms.net</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Complete Guide for Kubernetes Client-Go SharedInformerFactory]]></title>
            <link>https://levelup.gitconnected.com/a-complete-guide-for-kubernetes-client-go-sharedinformerfactory-db1e935ae745?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/db1e935ae745</guid>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[client-go]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 02:21:57 GMT</pubDate>
            <atom:updated>2025-07-14T02:21:57.071Z</atom:updated>
            <content:encoded><![CDATA[<h4>Master client-go factory patterns, custom indexing, and performance optimization techniques.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U5fomvuiCf9IBKG2t0HNnA.png" /><figcaption>client-go: sharedinformerfactory</figcaption></figure><p>Previously, we have explored how to get started with client-go and client-go informers in depth:</p><ul><li><a href="https://levelup.gitconnected.com/getting-started-with-kubernetes-client-go-9dacda6fffef"><strong>Getting Started With Kubernetes Client-Go</strong></a></li><li><a href="https://medium.com/p/012bb5362a38"><strong>A Deep Dive Into Kubernetes Client-Go Informers</strong></a></li></ul><p>We learned how <a href="https://medium.com/p/012bb5362a38"><strong><em>informers</em></strong></a> utilize the List-Watch pattern to efficiently monitor Kubernetes resources. However, we also discovered a major limitation: creating separate informers for each resource type leads to repetitive boilerplate code. We also discussed <a href="https://medium.com/gitconnected/a-deep-dive-into-kubernetes-client-go-informers-012bb5362a38#:~:text=Next-,part,-%3A"><strong><em>here</em></strong></a>.</p><pre>// Repetitive pattern for each resource type<br>func createPodInformer(clientset *kubernetes.Clientset) cache.SharedIndexInformer {<br>    // Boilerplate code for pods...<br>}<br><br>func createDeploymentInformer(clientset *kubernetes.Clientset) cache.SharedIndexInformer {<br>    // Almost identical boilerplate code for deployments...<br>}</pre><p>To prevent repeating the same pattern, we can use <strong><em>SharedInformerFactory</em></strong>, which allows us to manage multiple Kubernetes resources with a single factory.</p><h4>Significance of SharedInformerFactory</h4><p><strong><em>SharedInformerFactory</em></strong> is a factory pattern that creates and manages multiple informers efficiently. Instead of creating individual informers with separate connections, it coordinates resource sharing.</p><p><strong>Key Benefits:</strong></p><ul><li><strong>Reduces Boilerplate:</strong> Avoids repetitive code for creating informers.</li><li><strong>Efficient Resource Sharing:</strong> Informers share the same underlying connections and caches.</li><li><strong>Centralized Control:</strong> Easily start, stop, and synchronize all informers with a single line of code.</li><li><strong>Improved Performance:</strong> Minimizes memory usage and network overhead.</li></ul><pre>// One factory manages everything<br>factory := informers.NewSharedInformerFactory(clientset, time.Minute*10)<br><br>podInformer := factory.Core().V1().Pods()<br>deploymentInformer := factory.Apps().V1().Deployments()<br>serviceInformer := factory.Core().V1().Services()<br><br>factory.Start(stopCh)              // Start everything at once<br>factory.WaitForCacheSync(stopCh)   // Sync everything at once</pre><h4>How to Use SharedInformerFactory</h4><p>The following example demonstrates how multiple controllers share the same factory.</p><ul><li>GitHub: <a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/07_shared_informer_factory/main.go"><strong>Link</strong></a></li></ul><pre>package main<br><br>import (<br> &quot;flag&quot;<br> &quot;fmt&quot;<br> &quot;log&quot;<br> &quot;os&quot;<br> &quot;path/filepath&quot;<br> &quot;time&quot;<br><br> appsv1 &quot;k8s.io/api/apps/v1&quot;<br> corev1 &quot;k8s.io/api/core/v1&quot;<br> &quot;k8s.io/client-go/informers&quot;<br> &quot;k8s.io/client-go/kubernetes&quot;<br> &quot;k8s.io/client-go/tools/cache&quot;<br> &quot;k8s.io/client-go/tools/clientcmd&quot;<br>)<br><br><br>func main() {<br> // Create client<br> clientset := createClientSet()<br><br> // Single factory for all informers<br> factory := informers.NewSharedInformerFactory(clientset, time.Second*30)<br><br> // Setup multiple informers using same factory<br> setupPodMonitor(factory)<br> setupDeploymentMonitor(factory)<br> setupPodUpdateMonitor(factory)<br><br> // Start all informers at once<br> stopCh := make(chan struct{})<br> factory.Start(stopCh)<br> factory.WaitForCacheSync(stopCh)<br> &lt;-stopCh<br>}<br><br>// Controller 1: Pod Monitor<br>func setupPodMonitor(factory informers.SharedInformerFactory) {<br> podInformer := factory.Core().V1().Pods()<br><br> podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{<br>  AddFunc: func(obj interface{}) {<br>   pod := obj.(*corev1.Pod)<br>   fmt.Printf(&quot;[Monitor] Pod added: %s\n&quot;, pod.Name)<br>  },<br>  DeleteFunc: func(obj interface{}) {<br>   pod := obj.(*corev1.Pod)<br>   fmt.Printf(&quot;[Monitor] Pod deleted: %s\n&quot;, pod.Name)<br>  },<br> })<br>}<br><br>// Controller 2: Deployment Manager<br>// Deployment Manager by implementing cache.ResourceEventHandler<br>// Just showing another way to implement<br>type DeploymentHandler struct{}<br><br>func (h *DeploymentHandler) OnAdd(obj interface{}, isInInitialList bool) {<br> deployment := obj.(*appsv1.Deployment)<br> fmt.Printf(&quot;[Manager] Deployment added: %s\n&quot;, deployment.Name)<br>}<br><br>func (h *DeploymentHandler) OnUpdate(oldObj, newObj interface{}) {<br> // Implementation<br>}<br><br>func (h *DeploymentHandler) OnDelete(obj interface{}) {<br> // Implementation<br>}<br><br>func setupDeploymentMonitor(factory informers.SharedInformerFactory) {<br> deploymentInformer := factory.Apps().V1().Deployments()<br><br> handler := &amp;DeploymentHandler{}<br> deploymentInformer.Informer().AddEventHandler(handler)<br><br>}<br><br>// Controller 3: Pod Update Monitor (uses SAME Pod informer as Controller 1)<br>func setupPodUpdateMonitor(factory informers.SharedInformerFactory) {<br> podInformer := factory.Core().V1().Pods() // Gets the SAME shared Pod informer<br><br> podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{<br>  UpdateFunc: func(oldObj, newObj interface{}) {<br>   pod := newObj.(*corev1.Pod)<br>   fmt.Printf(&quot;[PodUpdateMonitor] Pod updated: %s\n&quot;, pod.Name)<br>   // logic here<br>  },<br> })<br>}<br><br>// createClientset creates and returns a Kubernetes clientset<br>func createClientSet() *kubernetes.Clientset {<br>   // Get home directory for kubeconfig path<br>   home, err := os.UserHomeDir()<br>   if err != nil {<br>    log.Fatalf(&quot;Failed to get home directory: %v&quot;, err)<br>   }<br>   // Parse kubeconfig flag<br>   kubeconfig := flag.String(&quot;kubeconfig&quot;, filepath.Join(home, &quot;/.kube/config&quot;), &quot;location of kubeconfig file&quot;)<br>   flag.Parse()<br>   // Build config from kubeconfig file<br>   config, err := clientcmd.BuildConfigFromFlags(&quot;&quot;, *kubeconfig)<br>   if err != nil {<br>    log.Fatalf(&quot;Failed to build config: %v&quot;, err)<br>   }<br>   // Create clientset<br>   clientset, err := kubernetes.NewForConfig(config)<br>   if err != nil {<br>    log.Fatalf(&quot;Failed to create clientset: %v&quot;, err)<br>   }<br>   return clientset<br>}</pre><p><strong>Key Insights: <br></strong>Both <strong>setupPodMonitor()</strong> and <strong>setupPodUpdateMonitor()</strong> use the same pod informer instance. They share the watch connection and cache but have separate event handlers.</p><pre>SharedInformerFactory<br>├── Pod Cache (shared by 2 controllers)<br>│   ├── setupPodMonitor handlers<br>│   └── setupPodUpdateMonitor handlers<br>│<br>└── Deployment Cache (used by 1 controller)<br>    └── setupDeploymentMonitor handlers</pre><h4>How SharedInformerFactory Works Internally</h4><p>The <strong><em>SharedInformerFactory</em></strong> keeps an internal registry of informers. When you call a method like factory.Core().V1().Pods(), the factory does the following:</p><ul><li>Checks if an informer for that resource type already exists.</li><li>If not, it creates a new informer and registers it internally.</li><li>If it does exist, it returns the existing informer.</li></ul><h4>What Happens When We Start the Factory</h4><p>When we call <strong>factory.Start(stopCh)</strong> , following things happen:</p><ul><li>It iterates through its internal registry of informers.</li><li>Only starts informers those who were registered before the Start()call.</li><li>Any informers created after Start()is called are ignored.</li></ul><p>For deeper insight into the informer factory, take a look at → <a href="https://github.com/kubernetes/client-go/blob/master/informers/factory.go#L143-L165C2"><strong><em>factory.go</em></strong></a></p><pre>type sharedInformerFactory struct {<br>    informers        map[reflect.Type]cache.SharedIndexInformer<br>    startedInformers map[reflect.Type]bool<br>    // ... other fields<br>}<br><br>func (f *sharedInformerFactory) Start(stopCh &lt;-chan struct{}) {<br> f.lock.Lock()<br> defer f.lock.Unlock()<br><br> if f.shuttingDown {<br>  return<br> }<br> <br> // Only start informers that are already registered &lt;--------<br> for informerType, informer := range f.informers { <br>  if !f.startedInformers[informerType] {<br>   f.wg.Add(1)<br>   informer := informer<br>   go func() {<br>    defer f.wg.Done()<br>    informer.Run(stopCh)<br>   }()<br>   f.startedInformers[informerType] = true<br>  }<br> }<br>}</pre><p>The key point is that <strong>Start()</strong>only iterates through <strong>f.informers</strong> that already existed at the time <strong>Start()</strong>is called.</p><h4>Understanding Informer Interfaces</h4><p>We have already observed the factory create an informer for a resource type. What’s also important is that it generates a <strong><em>lister</em></strong>, which we can use for local cache access and <strong><em>O(1) index lookups by namespace or labels</em></strong>.</p><p>Before diving into <strong><em>listers</em></strong>, let&#39;s understand the <strong><em>informer interfaces</em></strong> that the factory creates for each resource type.</p><p><strong>The PodInformer Interface: </strong><a href="https://github.com/kubernetes/client-go/blob/master/informers/core/v1/pod.go"><strong>Codebase</strong></a></p><pre>type PodInformer interface {<br>    Informer() cache.SharedIndexInformer  <br>    Lister() corev1.PodLister            <br>}<br><br>func (f *podInformer) Informer() cache.SharedIndexInformer {<br>    return f.factory.InformerFor(&amp;apicorev1.Pod{}, f.defaultInformer)<br>}<br><br>func (f *podInformer) Lister() corev1.PodLister {<br>    return corev1.NewPodLister(f.Informer().GetIndexer())<br>}</pre><p><strong>The DeploymentInformer Interface: </strong><a href="https://github.com/kubernetes/client-go/blob/master/informers/apps/v1/deployment.go"><strong>Codebase</strong></a></p><pre>type DeploymentInformer interface {<br>    Informer() cache.SharedIndexInformer   <br>    Lister() appsv1.DeploymentLister<br>}<br><br>func (f *deploymentInformer) Informer() cache.SharedIndexInformer {<br>    return f.factory.InformerFor(&amp;apiappsv1.Deployment{}, f.defaultInformer)<br>}<br><br>func (f *deploymentInformer) Lister() appsv1.DeploymentLister {<br>    return appsv1.NewDeploymentLister(f.Informer().GetIndexer())<br>}</pre><ul><li><strong>Informer()</strong>: For setup, event handling, and cache management</li><li><strong>Lister()</strong>: For fast, read-only access to cached data</li></ul><h4>Overall Workflow</h4><p>Here is the underlying workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3DQopf0HTLAASzE3WfKIA.png" /><figcaption>SharedInformerFactory</figcaption></figure><h4>Use Both Methods</h4><p>Below is an example demonstrating the use of an <strong><em>informer</em></strong> and <strong><em>lister</em></strong> together:</p><pre>func setupPodMonitor(factory informers.SharedInformerFactory) {<br>    podInformer := factory.Core().V1().Pods()<br>    <br>    // Method 1: Using Informer() - for setup and event handling<br>    sharedInformer := podInformer.Informer()<br>    sharedInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{<br>        AddFunc: func(obj interface{}) {<br>            pod := obj.(*corev1.Pod)<br>            fmt.Printf(&quot;Pod added: %s\n&quot;, pod.Name)<br>        },<br>    })<br>    <br>    // Method 2: Using Lister() - for querying cached data<br>    podLister := podInformer.Lister()<br>    pods, err := podLister.List(labels.Everything())<br>    if err != nil {<br>        fmt.Printf(&quot;Error: %v\n&quot;, err)<br>        return<br>    }<br>    fmt.Printf(&quot;Total pods: %d\n&quot;, len(pods))<br>}</pre><p>Here is an example code on how we can utilize <strong><em>lister</em></strong> for efficient query:</p><ul><li>GitHub: <a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/08_shared_informer_factory_lister/main.go#L73-L138C2"><strong>Link</strong></a></li></ul><pre><br>func setupInformers(factory informers.SharedInformerFactory) {<br>   // Create the informers so they get registered with the factory<br>   // Must call .Informer() to actually register them for syncing<br>   factory.Core().V1().Pods().Informer()<br>   factory.Apps().V1().Deployments().Informer()<br>}<br><br>func useListers(factory informers.SharedInformerFactory) {<br>   // Get listers<br>   podLister := factory.Core().V1().Pods().Lister()<br>   deploymentLister := factory.Apps().V1().Deployments().Lister()<br>  <br>   // Get ALL pods (across all namespaces)<br>   allPods, err := podLister.List(labels.Everything())<br>   if err != nil {<br>    fmt.Printf(&quot;Error: %v\n&quot;, err)<br>    return<br>   }<br>   fmt.Printf(&quot;Total pods (all namespaces): %d\n&quot;, len(allPods))<br>  <br>   // Show pods by namespace<br>   namespaceCount := make(map[string]int)<br>   for _, pod := range allPods {<br>    namespaceCount[pod.Namespace]++<br>   }<br>   fmt.Println(&quot;Pods per namespace:&quot;)<br>   for ns, count := range namespaceCount {<br>    fmt.Printf(&quot;  %s: %d pods\n&quot;, ns, count)<br>   }<br>  <br>   // Get pods in default namespace specifically<br>   defaultPods, err := podLister.Pods(&quot;default&quot;).List(labels.Everything())<br>   if err != nil {<br>    fmt.Printf(&quot;Error: %v\n&quot;, err)<br>    return<br>   }<br>   fmt.Printf(&quot;Pods in default namespace: %d\n&quot;, len(defaultPods))<br>  <br>   // Get specific pod by name (if any pods exist)<br>   if len(allPods) &gt; 0 {<br>    firstPod := allPods[0]<br>    pod, err := podLister.Pods(firstPod.Namespace).Get(firstPod.Name)<br>    if err != nil {<br>     fmt.Printf(&quot;Pod not found: %v\n&quot;, err)<br>    } else {<br>     fmt.Printf(&quot;Found pod: %s in namespace: %s\n&quot;, pod.Name, pod.Namespace)<br>    }<br>   }<br>  <br>   // Filter by labels<br>   labelSelector, _ := labels.Parse(&quot;app=nginx&quot;)<br>   nginxPods, err := podLister.List(labelSelector)<br>   if err != nil {<br>    fmt.Printf(&quot;Error: %v\n&quot;, err)<br>    return<br>   }<br>   fmt.Printf(&quot;Nginx pods: %d\n&quot;, len(nginxPods))<br>  <br>   // Query ALL deployments<br>   allDeployments, err := deploymentLister.List(labels.Everything())<br>   if err != nil {<br>    fmt.Printf(&quot;Error: %v\n&quot;, err)<br>    return<br>   }<br>   fmt.Printf(&quot;Total deployments (all namespaces): %d\n&quot;, len(allDeployments))<br>}</pre><h4>Custom Index Setup</h4><p>While <strong><em>listers</em></strong> allow filtering by namespaces and labels, we may need to filter by other fields like the <strong><em>node, pod image, </em></strong><em>or</em><strong><em> current phase</em></strong>. For these<br> scenarios, we must configure custom indexes and then query against them. The following example demonstrates how to set up and use a custom index:</p><ul><li>GitHub: <a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/09_shared_informer_factory_custom_index/main.go#L82-L140C2"><strong>Link</strong></a></li></ul><pre>// setupInformersWithCustomIndex creates Pod informer and adds custom indexes<br>func setupInformersWithCustomIndex(factory informers.SharedInformerFactory) {<br>   // Get Pod informer from factory (creates it if doesn&#39;t exist)<br>   podInformer := factory.Core().V1().Pods()<br>  <br>   // Add custom indexers to enable O(1) lookups by specific fields<br>   podInformer.Informer().AddIndexers(<br>    cache.Indexers{<br>     // Index pods by the node they&#39;re running on<br>     &quot;node&quot;: func(obj interface{}) ([]string, error) {<br>      pod := obj.(*corev1.Pod)<br>      return []string{pod.Spec.NodeName}, nil<br>     },<br>     // Index pods by their current phase (Running, Pending, etc.)<br>     &quot;phase&quot;: func(obj interface{}) ([]string, error) {<br>      pod := obj.(*corev1.Pod)<br>      return []string{string(pod.Status.Phase)}, nil<br>     },<br>     // Additional custom indexes can be added here<br>    })<br>}<br><br>// queryWithCustomIndexers demonstrates how to use custom indexes for efficient queries<br>func queryWithCustomIndexers(factory informers.SharedInformerFactory) {<br>   // Get the indexer from Pod informer to perform custom queries<br>   indexer := factory.Core().V1().Pods().Informer().GetIndexer()<br>  <br>   // Query 1: Get all unique node names that have pods<br>   allNodes := indexer.ListIndexFuncValues(&quot;node&quot;)<br>   fmt.Printf(&quot;Available nodes: %v\n&quot;, allNodes)<br>  <br>   // Query 2: Get all pods on the first available node<br>   if len(allNodes) &gt; 0 {<br>    nodeName := allNodes[0]<br>    // Use custom &quot;node&quot; index for O(1) lookup<br>    podsOnNode, err := indexer.ByIndex(&quot;node&quot;, nodeName)<br>    if err != nil {<br>     fmt.Printf(&quot;Error: %v\n&quot;, err)<br>     return<br>    }<br>    fmt.Printf(&quot;Pods on node &#39;%s&#39;: %d\n&quot;, nodeName, len(podsOnNode))<br>  <br>    // Extract and display pod names and namespaces<br>    for _, obj := range podsOnNode {<br>     pod := obj.(*corev1.Pod)<br>     fmt.Printf(&quot;  - %s (namespace: %s)\n&quot;, pod.Name, pod.Namespace)<br>    }<br>   } else {<br>    fmt.Println(&quot;No nodes found&quot;)<br>   }<br>  <br>   // Query 3: Get all pods in &quot;Running&quot; phase using custom index<br>   runningPods, err := indexer.ByIndex(&quot;phase&quot;, &quot;Running&quot;)<br>   if err != nil {<br>    fmt.Printf(&quot;Error: %v\n&quot;, err)<br>    return<br>   }<br>   fmt.Printf(&quot;Running pods: %d\n&quot;, len(runningPods))<br>}</pre><h4>Create Factory With Other Options:</h4><p>There are several options available for creating a <strong><em>SharedInformerFactory</em></strong>. We can scope it to a specific namespace, filter by resource labels, and even configure different <a href="https://levelup.gitconnected.com/a-deep-dive-into-kubernetes-client-go-informers-012bb5362a38#:~:text=of%20safe%20storage-,Resync%20TimePeriod,-While%20creating%20an"><strong><em>resync</em></strong></a> intervals based on the resource type.</p><ul><li><strong><em>Based on Namespace:</em></strong></li></ul><pre> // Create factory scoped to specific namespace<br> informers.NewSharedInformerFactoryWithOptions(<br>  clientset,<br>  time.Second*30,<br>  informers.WithNamespace(&quot;kube-system&quot;),<br> )</pre><ul><li><strong><em>Based on the label:</em></strong></li></ul><pre> // Create factory filtered by label selector<br> informers.NewSharedInformerFactoryWithOptions(<br>  clientset,<br>  time.Second*30,<br>  informers.WithTweakListOptions(func(lo *metav1.ListOptions) {<br>   lo.LabelSelector = &quot;app=nginx&quot;<br>  }),<br> )</pre><ul><li><strong><em>Different resync times based on resource type:</em></strong></li></ul><pre>// Create factory with custom resync periods per resource type: <br>informers.NewSharedInformerFactoryWithOptions(<br>  clientset,<br>  time.Second*30,<br>  informers.WithCustomResyncConfig(map[metav1.Object]time.Duration{<br>   &amp;corev1.Pod{}:        time.Second * 10,<br>   &amp;appsv1.Deployment{}: time.Second * 60,<br>  }),<br> )</pre><p>For other approaches to creating a factory, please refer to the examples available on here → <a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/11_shared_informer_factory_with_options/main.go"><strong>GitHub</strong></a>.</p><h4>Next Part:</h4><p>We’ve seen how <strong><em>SharedInformerFactory</em></strong> helps to manage multiple Kubernetes resources efficiently by sharing connections and caches. However, it also comes with a limitation: <strong>events</strong> in the <strong>informer handlers</strong> are processed in <strong>real time</strong>, which can become a <strong>bottleneck</strong>.</p><pre>// Problematic direct processing in event handlers<br>podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{<br>    AddFunc: func(obj interface{}) {<br>        pod := obj.(*corev1.Pod)<br>        <br>        // PROBLEM: Heavy processing blocks the informer<br>        if err := heavyProcessingOperation(pod); err != nil {<br>            // PROBLEM: Error is lost, no retry mechanism<br>            fmt.Printf(&quot;Failed to process pod: %v\n&quot;, err)<br>        }<br>    },<br>})</pre><ul><li><strong>Blocking Operations: </strong>Heavy processing in<strong> event handlers</strong> blocks the informer.</li><li><strong>Resource Exhaustion: </strong>Burst events can overwhelm the system.</li><li><strong>No Rate Limiting:</strong> Rapid-fire events can cause API server throttling.</li></ul><p>To address these challenges, we need a <strong>WorkQueue</strong> — a reliable asynchronous processing system that separates event detection from event handling. In the next part, we’ll dive into client-go <strong>WorkQueue</strong>.</p><h4>✅<strong> All code related to client-go can be found here: </strong><a href="https://github.com/shamimice03/mastering-k8s-client-go/tree/main"><strong>GitHub</strong></a></h4><h4>✍️<strong> All articles related to client-go:</strong></h4><p><a href="https://medium.com/@shamimice03/list/af35336501e4">Kubernetes Client-Go Series</a></p><h4>☸️ All articles related to kubernetes:</h4><p><a href="https://medium.com/@shamimice03/list/7ae1a0f96f3b">All Articles on Kubernetes</a></p><blockquote><em>If you found this article useful, please click on the Follow 👉 and Clap 👏 buttons to support writing </em><strong><em>more about kubernetes client-go.</em></strong><em><br>Thank you ❤</em><br>🔔<em> Find me on</em><strong><em> </em></strong><a href="https://www.linkedin.com/in/shamimice03/"><strong><em>LinkedIn </em></strong></a><strong><em>| </em></strong><a href="http://github.com/shamimice03"><strong><em>GitHub </em></strong></a><strong><em>| </em></strong><a href="https://twitter.com/shamimice03"><strong><em>Twitter</em></strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db1e935ae745" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/a-complete-guide-for-kubernetes-client-go-sharedinformerfactory-db1e935ae745">A Complete Guide for Kubernetes Client-Go SharedInformerFactory</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deep Dive Into Kubernetes Client-Go Informers]]></title>
            <link>https://levelup.gitconnected.com/a-deep-dive-into-kubernetes-client-go-informers-012bb5362a38?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/012bb5362a38</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[client-go]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 01:31:49 GMT</pubDate>
            <atom:updated>2025-06-30T01:31:49.491Z</atom:updated>
            <content:encoded><![CDATA[<h4>Step-by-step guide to implementing List-Watch pattern, custom indexers, and shared event handlers in Kubernetes Client-Go</h4><p>Previously, we explored <a href="https://medium.com/gitconnected/getting-started-with-kubernetes-client-go-9dacda6fffef"><strong>how to create Kubernetes resources</strong> </a>using client-go. Now, we’ll dive into <strong>client-go informers </strong>— a key feature of client-go essential for developing custom Kubernetes controllers and operators.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ojhvo7IHbiGjXnTlKZvOhg.png" /><figcaption>How informer works.</figcaption></figure><h3><strong>What is Informer?</strong></h3><p>Informers are a core component of client-go and play a vital role when developing Kubernetes controllers or operators. They enable efficient local watching and caching of Kubernetes resources. Without informers, we would need to constantly query the Kubernetes API server — an approach that quickly becomes inefficient.</p><p>For example, if we’re trying to monitor the status of pods in the default namespace, we might end up doing something like this:</p><ul><li><strong>GitHub →</strong> <a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/03_without_informer/main.go#L14C1-L22C2"><strong>Link</strong></a></li></ul><pre>func podStatus(clientset *kubernetes.Clientset) {<br>   for {<br>    pods, _ := clientset.CoreV1().Pods(&quot;default&quot;).List(context.TODO(), metav1.ListOptions{})<br>    for _, pod := range pods.Items {<br>     fmt.Printf(&quot;%s: %s\n&quot;, pod.Name, pod.Status.Phase)<br>    }<br>    fmt.Println(&quot;---&quot;)<br>   }<br>}</pre><p><strong>Output</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b7nFT4KEPrCgce_vwLzIXw.png" /></figure><p>In client-go, the informer monitors changes in Kubernetes resources and caches the resource state locally, significantly reducing the need for direct Kubernetes API calls.</p><h3>How Informer Works?</h3><p>Informers do not follow the polling pattern. Instead, it follows the <em>List-Watch</em> pattern. The diagram below illustrates how an informer collects and manages information about pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Otu5MpEsMEALGtM51Ee8DA.png" /><figcaption>How Clinet-go Informers Works</figcaption></figure><p>It&#39;s similar to this:</p><pre>GET /api/v1/namespaces/default/pods?watch=true</pre><p>The ?watch=true parameter creates a <strong>long-lived HTTP streaming connection</strong>.</p><pre>&gt;&gt; GET /api/v1/namespaces/default/pods?watch=true HTTP/1.1<br><br>Host: kubernetes-api-server:6443<br>Authorization: Bearer &lt;token&gt;<br>Accept: application/json<br>Connection: keep-alive<br><br>HTTP/1.1 200 OK<br>Content-Type: application/json<br>Transfer-Encoding: chunked<br>Connection: keep-alive<br><br>{&quot;type&quot;:&quot;ADDED&quot;,&quot;object&quot;:{&quot;metadata&quot;:{&quot;name&quot;:&quot;pod-1&quot;},...}}<br>{&quot;type&quot;:&quot;MODIFIED&quot;,&quot;object&quot;:{&quot;metadata&quot;:{&quot;name&quot;:&quot;pod-1&quot;},...}}<br>{&quot;type&quot;:&quot;DELETED&quot;,&quot;object&quot;:{&quot;metadata&quot;:{&quot;name&quot;:&quot;pod-2&quot;},...}}<br>{&quot;type&quot;:&quot;ADDED&quot;,&quot;object&quot;:{&quot;metadata&quot;:{&quot;name&quot;:&quot;pod-3&quot;},...}}<br>...</pre><p>With this <strong>long-lived HTTP streaming connection, </strong>any changes in the resources will be pushed by the API server to the informers who are watching.</p><p>It’s also possible to verify this watch pattern from our local:</p><pre># Terminal 1: Start kubectl proxy<br>kubectl proxy --port=8080<br><br># Terminal 2: Test the watch API with curl<br>curl -s &quot;http://localhost:8080/api/v1/namespaces/default/pods?watch=true&quot;</pre><p>Now, if we create a new pod, we will see streaming JSON events in the terminal in real-time.</p><h3>How does the API server detect changes?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ojhvo7IHbiGjXnTlKZvOhg.png" /><figcaption>How the API Server is aware of the changes</figcaption></figure><ul><li><strong>Change</strong>: kubectl/controllers sends HTTP requests to the API server.</li><li><strong>Storage/etcd</strong>: The API server stores changes in the <strong>etcd</strong> key-value store.</li><li><strong>etcd</strong>: etcd notifies the API server when data changes using its built-in watch mechanism.</li><li><strong>Broadcast</strong>: API server sends events to all watching informers via persistent HTTP connections.</li></ul><h3>Creating Our First Informer</h3><p>In this section, we’ll create an informer for Pods. To do that, we need to follow the pattern outlined below.</p><pre>module (<br>   corev1 &quot;k8s.io/api/core/v1&quot;<br>  &quot;k8s.io/client-go/tools/cache&quot;<br>)<br><br>// Creates watch connection + local cache<br>informer := cache.NewSharedIndexInformer(<br>    &amp;cache.ListWatch{<br>        ListFunc:  // Gets all pods initially<br>        WatchFunc: // Watches for changes<br>    },<br>    &amp;corev1.Pod{},  // &lt;- For pods<br>    resyncPeriod,<br>    indexers,<br>)</pre><p>Similarly, we can create an informer for other kubernetes resources as well. The following is the function to create a <strong>PodInformer</strong>.</p><ul><li><strong>GitHub →</strong><a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/04_informer_events/main.go#L44C1-L63C2"><strong> Link</strong></a></li></ul><pre>// createPodInformer creates and returns a SharedIndexInformer for pods<br>func createPodInformer(clientset *kubernetes.Clientset) cache.SharedIndexInformer {<br><br> // Create SharedIndexInformer with ListWatch functions<br> informer := cache.NewSharedIndexInformer(<br>  &amp;cache.ListWatch{<br>   // List function - gets initial state of pods<br>   ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {<br>    return clientset.CoreV1().Pods(&quot;&quot;).List(context.TODO(), options)<br>   },<br>   // Watch function - creates streaming connection for pod changes<br>   WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {<br>    return clientset.CoreV1().Pods(&quot;&quot;).Watch(context.TODO(), options)<br>   },<br>  },<br>  &amp;corev1.Pod{},    // Object type to watch<br>  time.Second*30,   // Resync period<br>  cache.Indexers{}, // Custom indexers<br> )<br><br> return informer<br>}</pre><h3>Understanding the SharedIndexInformer</h3><p>As we’ve seen in the code, informers rely on the List-Watch pattern to monitor Kubernetes resources. Now, let’s explore the core component behind this functionality: <strong>SharedIndexInformer</strong>.</p><p>Recall our earlier example of inefficient polling? Now, let’s take a look at how <strong>SharedIndexInformer</strong> addresses this issue:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OANs3i2VRNzkdMasdgH0lQ.png" /><figcaption>How SharedIndexInformer works</figcaption></figure><h4>The Three Aspects of SharedIndexInformer</h4><p>The name SharedIndexInformer reflects three key capabilities combined into one:</p><ul><li><strong>Shared</strong>: Multiple event handlers can use the same informer instance. In the above part, we created a <em>PodInformer</em> that can be used by multiple event handlers.</li><li><strong>Index</strong>: Fast lookups using custom indexes.</li><li><strong>Informer</strong>: The core watching and caching functionality.</li></ul><h4>Shared: Multiple Event Handlers</h4><p>Multiple controllers can register their event handlers with the same informer instance, sharing the underlying watch connection and cache.</p><pre>// One informer instance<br>podInformer := createPodInformer(clientset)<br><br>// Multiple handlers share the same informer<br>podInformer.AddEventHandler(podWatcherHandler)    // Handler 1<br>podInformer.AddEventHandler(resourceQuotaHandler) // Handler 2 <br>podInformer.AddEventHandler(networkHandler)       // Handler 3<br><br>// Only ONE watch connection to API server for all handlers<br>go podInformer.Run(stopCh)</pre><p><strong>Key Points: </strong>Instead of each handler creating its connection, they all share one connection and cache.</p><p>The following is an example of how multiple handlers can consume the same <strong>podInformer</strong>.</p><ul><li><strong>GitHub → </strong><a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/04_informer_events/main.go#L84C1-L108C1"><strong>Link</strong></a></li></ul><pre> // SHARED ASPECT: First handler - multiple handlers can share the same informer<br> podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{<br>  AddFunc: func(obj interface{}) {<br>   pod := obj.(*corev1.Pod)<br>   fmt.Printf(&quot;(+) Pod added: %s/%s\n&quot;, pod.Namespace, pod.Name)<br>  },<br>  UpdateFunc: func(oldObj, newObj interface{}) {<br>   pod := newObj.(*corev1.Pod)<br>   fmt.Printf(&quot;(*) Pod updated: %s/%s\n&quot;, pod.Namespace, pod.Name)<br>  },<br>  DeleteFunc: func(obj interface{}) {<br>   pod := obj.(*corev1.Pod)<br>   fmt.Printf(&quot;(-) Pod deleted: %s/%s\n&quot;, pod.Namespace, pod.Name)<br>  },<br> })<br><br> // SHARED ASPECT: Second handler on the SAME informer instance<br> // Both handlers share the same watch connection and cache<br> podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{<br>  AddFunc: func(obj interface{}) {<br>   pod := obj.(*corev1.Pod)<br>   fmt.Printf(&quot;[SECOND-CONTROLLER] Also saw pod: %s/%s\n&quot;, pod.Namespace, pod.Name)<br>  },<br> })<br>    </pre><p>However, this implementation is limited to pod resources. For other resources, like deployments, a separate informer — such as a <strong>deploymentInformer</strong> — must be created. These limitations can be overcome by using a factory method to generate informers, which we will explore as we proceed.</p><h4>Indexed: O(1) Lookup for resources</h4><p>When building a kubernetes controller or operator, we may need to request resources running on a specific namespace or nodes. Without indexing, this requires scanning through ALL cached pods each time — a highly inefficient O(n) operation.</p><ul><li><strong>GitHub → </strong><a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/05_informer_index/main.go#L89C1-L98C3"><strong>Link</strong></a></li></ul><p>The following is an example of an O(n) solution:</p><pre><br> // Inefficient: O(n) search through all pods without indexing<br> // Get stores and list all cached objects<br> podStore := podInformer.GetStore()<br> allPods := podStore.List()<br><br> fmt.Printf(&quot;Found %d pods\n&quot;, len(allPods))<br><br> // Print pod details<br> for _, obj := range allPods {<br>  pod := obj.(*corev1.Pod)<br>  fmt.Printf(&quot;Pod: %s/%s \nNode: %s\n\n&quot;, pod.Namespace, pod.Name, pod.Spec.NodeName)<br> }</pre><p>With indexing, we can get <strong>O(1) lookups, </strong>just like that:</p><pre>indexer := podInformer.GetIndexer()<br>defaultPods, err := indexer.ByIndex(&quot;namespace&quot;, &quot;default&quot;)</pre><p>Resources can be indexed in several ways. The following is an example of how <strong><em>pods</em></strong> are indexed based on <strong><em>namespace</em></strong> and <strong><em>nodes</em></strong>.</p><ul><li><strong>GitHub</strong> → <a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/05_informer_index/main.go#L44C1-L72C2"><strong>Link</strong></a></li></ul><pre>// Custom indexer function: extracts node name from pod for indexing<br>func podNodeIndexFunc(obj interface{}) ([]string, error) {<br>   pod := obj.(*corev1.Pod)<br>   return []string{pod.Spec.NodeName}, nil<br>}<br><br>// createPodInformer creates and returns a SharedIndexInformer for pods<br>func createPodInformer(clientset *kubernetes.Clientset) cache.SharedIndexInformer {<br>   informer := cache.NewSharedIndexInformer(<br>    &amp;cache.ListWatch{<br>         ....<br>       },<br>         ....<br>       },<br>    },<br>    &amp;corev1.Pod{},  <br>    time.Second*30, <br>    cache.Indexers{<br>     &quot;namespace&quot;: cache.MetaNamespaceIndexFunc, // for namespace index<br>     &quot;node&quot;:      podNodeIndexFunc,  // for node index, custom func needed<br>    }, <br> )<br> return informer<br>}</pre><p>The informer stores indexes like the following:</p><pre>namespace index:<br>  &quot;default&quot; → [pod1, pod2, pod3]<br>  &quot;kube-system&quot; → [pod4, pod5]<br>  <br>node index:  <br>  &quot;node-1&quot; → [pod1, pod6]<br>  &quot;node-2&quot; → [pod2]</pre><p>The following code demonstrates how to efficiently access <strong><em>pods</em></strong> filtered by <strong><em>namespace</em></strong> and <strong><em>node</em></strong>.</p><ul><li><strong>GitHub</strong> → <a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/05_informer_index/main.go#L100C1-L130C3"><strong>Link</strong></a></li></ul><pre>// NAMESPACE <br>// Efficient: O(1) lookup using namespace index<br> fmt.Println(&quot;\n=== With Namespace Index ===&quot;)<br> indexer := podInformer.GetIndexer()<br> defaultPods, err := indexer.ByIndex(&quot;namespace&quot;, &quot;default&quot;)<br> if err != nil {<br>  fmt.Printf(&quot;Error getting indexed values: %v\n&quot;, err)<br> }<br> fmt.Printf(&quot;Pods in default namespace: %d\n&quot;, len(defaultPods))<br><br> // Extract pod details from namespace index results<br> fmt.Println(&quot;Pod details from namespace index:&quot;)<br> for _, obj := range defaultPods {<br>  pod := obj.(*corev1.Pod)<br>  fmt.Printf(&quot;  Name: %s, Namespace: %s, Node: %s\n&quot;,<br>   pod.Name, pod.Namespace, pod.Spec.NodeName)<br> }<br> <br> // NODE<br> // Efficient: O(1) lookup using custom node index<br> fmt.Println(&quot;\n=== With Node Index ===&quot;)<br> nodeName := &quot;k3s-cloudterms-k8s-1486-8a8686-node-pool-c68e-kited&quot;<br> podsOnNode, err := indexer.ByIndex(&quot;node&quot;, nodeName)<br> if err != nil {<br>  fmt.Printf(&quot;Error getting indexed values: %v\n&quot;, err)<br> }<br><br> // Extract pod details from node index results<br> fmt.Printf(&quot;Pods on node %s: %d\n&quot;, nodeName, len(podsOnNode))<br> for _, obj := range podsOnNode {<br>  pod := obj.(*corev1.Pod)<br>  fmt.Printf(&quot;  Name: %s, Namespace: %s\n&quot;, pod.Name, pod.Namespace)<br> }</pre><h4>Informer: Watches and Caches</h4><p>As we already discussed, the Informer uses a <strong><em>List-Watch</em></strong> approach to retrieve, monitor, and locally cache objects. It depends heavily on caching mechanisms to function effectively.</p><h4>Internals of Informer</h4><p>To maintain a streaming connection with the API server while caching objects locally. This functionality relies on the following key components:</p><ul><li>Reflector</li><li>Delta FIFO Queue</li><li>Indexer</li><li>Thread-safe Store</li></ul><p>The following is the diagram of how the connection between the informer and the API server works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tkRPCBJH6VPSXJQpHWEoQ.png" /><figcaption>Image inspired from here: <a href="https://github.com/kubernetes/sample-controller/blob/master/docs/images/client-go-controller-interaction.jpeg">Here</a></figcaption></figure><h4>Reflector:</h4><p>Reflector is the component that receives and handles the streaming connection from the API server.</p><p>Implementation: <a href="https://github.com/kubernetes/client-go/blob/master/tools/cache/reflector.go">Link</a></p><p><strong>Execution flow:</strong></p><p>After starting the <strong>podInformers</strong>, the following is the flow of triggering the reflector</p><blockquote><em>SharedIndexInformer.Run() → controller.RunWithContext() (from </em><a href="https://github.com/kubernetes/client-go/blob/master/tools/cache/shared_informer.go"><em>sharedinformer.go</em></a><em>)</em></blockquote><blockquote><em>controller.RunWithContext() → starts reflector (from </em><a href="https://github.com/kubernetes/client-go/blob/master/tools/cache/controller.go"><em>controller.go</em></a><em>)</em></blockquote><h4>DeltaFIFO:</h4><ul><li>Temporarily stores events from Reflector before processing.</li><li>Maintains an ordered queue of changes (added, updated, or deleted).</li><li>Prevents blocking by allowing Reflector to continue receiving events during processing.</li></ul><h4>Indexer and thread-safe store:</h4><p>Informer itself doesn’t store objects — it just orchestrates the process:</p><ul><li>Informer receives objects from DeltaFIFO.</li><li>Informer puts objects into the Indexer/ThreadSafeStore.</li><li>Informer notifies event handlers.</li></ul><blockquote><em>ThreadSafeStore = Safe concurrent storage<br>Indexer = Fast lookups on top of safe storage</em></blockquote><h3>Resync TimePeriod</h3><p>While creating an informer, we have passed a parameter called resyncPeriod . Resync is a local cache operation in the kubernetes informer that periodically triggers event handlers without calling the API server.</p><pre>module (<br>   corev1 &quot;k8s.io/api/core/v1&quot;<br>  &quot;k8s.io/client-go/tools/cache&quot;<br>)<br><br>informer := cache.NewSharedIndexInformer(<br>    &amp;cache.ListWatch{<br>        ListFunc:  <br>        WatchFunc: <br>    },<br>    &amp;corev1.Pod{}, <br>    time.Second*30, // &lt;- Resync period - triggers every 30 seconds<br>    indexers,<br>)</pre><p>Resync is not about getting fresh data from the API server — it’s about giving our controller periodic opportunities to ensure everything is in the correct state based on what’s already cached locally.</p><p>Resync only triggers the <strong>UpdateFunc</strong>. If the cached data has changed, we’ll see different resource versions; if the data is unchanged, <strong>oldObj</strong> and <strong>newObj</strong> will be identical.</p><p>With the following code, we can catch the resync events:</p><ul><li><strong>GitHub → </strong><a href="https://github.com/shamimice03/mastering-k8s-client-go/blob/main/06_Resync/main.go#L84C1-L108C4"><strong>Link</strong></a></li></ul><pre>UpdateFunc: func(oldObj, newObj interface{}) {<br>   oldPod := oldObj.(*corev1.Pod)<br>   newPod := newObj.(*corev1.Pod)<br><br>   fmt.Printf(&quot;(*) Pod updated: %s/%s\n&quot;, newPod.Namespace, newPod.Name)<br>   fmt.Printf(&quot;Old pod version %v\n&quot;, oldPod.ResourceVersion)<br>   fmt.Printf(&quot;New pod version %v\n&quot;, newPod.ResourceVersion)<br>   <br>   // checking old and new version<br>   if oldPod.ResourceVersion == newPod.ResourceVersion {<br>    fmt.Printf(&quot;Resync event (same ResourceVersion)\n&quot;)<br>   } else {<br>    fmt.Printf(&quot;Real update event\n&quot;)<br>   }<br><br> }</pre><h3>Next part:</h3><p>In this blog, we’ve learned that to watch Kubernetes resources like pods, we must set up an informer specifically for pod resources. Likewise, to monitor the status of Deployments, a separate informer for deployments is required. The following code illustrates this:</p><pre>func createDeploymentInformer(clientset *kubernetes.Clientset) cache.SharedIndexInformer {<br> informer := cache.NewSharedIndexInformer(<br>  &amp;cache.ListWatch{<br>   ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {<br>    return clientset.AppsV1().Deployments(&quot;&quot;).List(context.TODO(), options)<br>   },<br>   WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {<br>    return clientset.AppsV1().Deployments(&quot;&quot;).Watch(context.TODO(), options)<br>   },<br>  },<br>  &amp;appsv1.Deployment{}, // Object type to watch<br>  time.Second*30,       // Resync period<br>  cache.Indexers{},     // Custom indexers<br> )<br> return informer<br>}</pre><p>If we observe, we can see codes are almost identical to PodInformer. To eliminate this kind of repetitive boilerplate, we can use the <strong><em>SharedInformerFactory</em></strong>. In the next blog, we’ll take a deeper dive into how it <strong><em>SharedInformerFactory</em></strong> works.</p><blockquote><em>If you found this article useful, please click on the Follow 👉 and Clap 👏 buttons to support writing </em><strong><em>more about kubernetes client-go.</em></strong><em><br>Thank you ❤</em><br>🔔<em> Find me on</em><strong><em> </em></strong><a href="https://www.linkedin.com/in/shamimice03/"><strong><em>LinkedIn </em></strong></a><strong><em>| </em></strong><a href="http://github.com/shamimice03"><strong><em>GitHub </em></strong></a><strong><em>| </em></strong><a href="https://twitter.com/shamimice03"><strong><em>Twitter</em></strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=012bb5362a38" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/a-deep-dive-into-kubernetes-client-go-informers-012bb5362a38">A Deep Dive Into Kubernetes Client-Go Informers</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started With Kubernetes Client-Go]]></title>
            <link>https://levelup.gitconnected.com/getting-started-with-kubernetes-client-go-9dacda6fffef?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dacda6fffef</guid>
            <category><![CDATA[kubernetes-api]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[client-go]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 15:12:57 GMT</pubDate>
            <atom:updated>2025-06-04T15:12:57.553Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding Kubernetes API structures and creating your first deployment using client-go</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ze9p6QufH0Bq3PR2UJiWww.png" /><figcaption>Inside client-go: Deployment resource architecture</figcaption></figure><h3>EntryPoint Point</h3><p>If we run <strong>kubectl create deploy nginx --image=nginx:1.21 </strong>Then what will be the result? — Kubernetes will magically create a deployment for us, right? And under that deployment, we have <strong><em>replica sets</em></strong> and <strong><em>pods</em></strong>. But what if we want to do this programmatically? And that is where client-go comes into the picture.</p><h3>Client-Go Intro</h3><p>Using <a href="https://github.com/kubernetes/client-go"><strong>client-go</strong></a>,<strong> </strong>we can programmatically access the Kubernetes API. But it’s built on top of other foundational Kubernetes modules. Two of the major kubernetes modules are:</p><blockquote><a href="https://github.com/kubernetes/api"><strong><em>k8s.io/api</em></strong></a><em> — Provide kubernetes resource definitions.<br></em><a href="https://github.com/kubernetes/apimachinery"><strong><em>k8s.io/apimachinery</em></strong></a><em> — Provides foundational building blocks.</em></blockquote><h4><strong>📌 Here’s how they are connected with </strong><a href="https://github.com/kubernetes/client-go"><strong>client-go</strong></a></h4><p>✔ <a href="https://github.com/kubernetes/api"><strong>k8s.io/api</strong></a><strong>:</strong> Specifies how Kubernetes resources (Deployments, Pods, Services) are structured. <br>See here: <a href="https://github.com/kubernetes/api/blob/master/apps/v1/types.go">types.go</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3tKQ6bo9HcXlom4hMhOfQ.png" /><figcaption><a href="https://github.com/kubernetes/api/blob/master/apps/v1/types.go">https://github.com/kubernetes/api/blob/master/apps/v1/types.go</a></figcaption></figure><p>✔ <a href="https://github.com/kubernetes/apimachinery"><strong>k8s.io/apimachinery</strong></a><strong>:</strong></p><p>Provides the underlying infrastructure components like <strong><em>ObjectMeta</em></strong>, <strong><em>TypeMeta</em></strong>, <strong><em>labels</em></strong>, and <strong><em>selectors</em></strong> that power all Kubernetes resources.</p><p>See here: <a href="https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go#L42">types.go</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zO3clywrs9cyE5mHUc1oRg.png" /><figcaption><a href="https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go#L42">https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go#L42</a></figcaption></figure><h4><strong>📌 The key insight:</strong></h4><p>If we observe closely, the <a href="https://github.com/kubernetes/api"><strong>k8s.io/api</strong></a><strong> </strong>repo doesn&#39;t define these fundamental types—it only imports them from <a href="https://github.com/kubernetes/apimachinery"><strong>k8s.io/apimachinery</strong></a>.</p><ul><li>Every Kubernetes object (Pod, Service, Deployment) has the same <strong><em>TypeMeta</em></strong> and <strong><em>ObjectMeta</em></strong> structure.</li><li>The <a href="https://github.com/kubernetes/api"><strong>k8s.io/api</strong></a><strong> </strong>repo focuses on resource-specific fields (like <strong><em>DeploymentSpec </em></strong><em>and</em><strong><em> </em></strong><strong><em>DeploymentStatus.</em></strong></li><li>The <a href="https://github.com/kubernetes/apimachinery"><strong>k8s.io/apimachinery</strong></a><strong> </strong>repo provides the universal building blocks that make every Kubernetes object consistent.</li></ul><blockquote><strong>apimachinery</strong> — &quot;Every Kubernetes object needs these basic fields&quot;<br><strong>api</strong> — &quot;Here&#39;s how to combine those basic fields into specific resources&quot;</blockquote><h4>✔<a href="https://github.com/kubernetes/client-go"><strong> client-go</strong></a></h4><p>Gives us the tools to manipulate Kubernetes resources and objects.</p><p><strong>Note:</strong> In <a href="https://github.com/kubernetes/client-go"><strong>client-go</strong></a>, we have multiple packages that serve different purposes for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aWAu-N7hiQ700rpzoEQFXQ.png" /><figcaption>Reference: <a href="https://github.com/kubernetes/client-go/tree/master?tab=readme-ov-file#whats-included">Link</a></figcaption></figure><h4>📌 Example of <a href="https://github.com/kubernetes/client-go/tree/master/kubernetes"><strong>client-go/kubernetes</strong></a> package usage:</h4><pre>// Import the client-go library<br>import &quot;k8s.io/client-go/kubernetes&quot;<br><br>// Create a clientset to interact with Kubernetes API<br>clientset, err := kubernetes.NewForConfig(config)<br><br>// Use the clientset to create a deployment<br>clientset.AppsV1().Deployments(&quot;default&quot;).Create(...)</pre><h3>Interact with the kubernetes API</h3><p>The following code demonstrates how to list all pods running in the <strong>default</strong> namespace.</p><ul><li>Make sure the Kubernetes cluster is up and running.</li><li>Ensure the kubeconfig file is available at: <strong>~/.kube/config.</strong></li><li>For implementation details, refer to the comments within the code.</li></ul><p>📂 <strong>GitHub:</strong> <a href="https://github.com/shamimice03/mastering-k8s-client-go/tree/main/01_starter/k8s-api-access">k8s-api-access</a></p><pre>package main<br><br>import (<br> &quot;context&quot;<br> &quot;fmt&quot;<br> &quot;path/filepath&quot;<br><br> // k8s.io/api - Kubernetes resource definitions<br> // Contains all the Kubernetes API objects like Pod, Service, Deployment, etc.<br><br> // k8s.io/apimachinery - Common building blocks<br> // Provides meta types and utilities used across Kubernetes APIs<br> metav1 &quot;k8s.io/apimachinery/pkg/apis/meta/v1&quot;<br><br> // k8s.io/client-go - Client library for Kubernetes API<br> // Main client library for interacting with Kubernetes clusters<br> &quot;k8s.io/client-go/kubernetes&quot;<br> &quot;k8s.io/client-go/rest&quot;<br> &quot;k8s.io/client-go/tools/clientcmd&quot;<br> &quot;k8s.io/client-go/util/homedir&quot;<br>)<br><br>// getExternalClusterConfig loads kubeconfig from ~/.kube/config<br>// This function is used to connect to external Kubernetes clusters<br>// by reading the standard kubectl configuration file<br>func getExternalClusterConfig() (*rest.Config, error) {<br> var kubeconfig string<br><br> // Determine the path to the kubeconfig file<br> // Typically located at ~/.kube/config (standard kubectl location)<br> if home := homedir.HomeDir(); home != &quot;&quot; {<br>  kubeconfig = filepath.Join(home, &quot;.kube&quot;, &quot;config&quot;)<br> }<br><br> // Build config from kubeconfig file<br> // This parses the YAML kubeconfig and creates a rest.Config object<br> // The first parameter is for master URL override (empty means use kubeconfig)<br> config, err := clientcmd.BuildConfigFromFlags(&quot;&quot;, kubeconfig)<br> if err != nil {<br>  return nil, fmt.Errorf(&quot;failed to build config from kubeconfig: %v&quot;, err)<br> }<br><br> return config, nil<br>}<br><br>func main() {<br> // Get external cluster configuration<br> // This establishes connection parameters to the Kubernetes API server<br> config, err := getExternalClusterConfig()<br> if err != nil {<br>  panic(fmt.Errorf(&quot;failed to get external cluster config: %v&quot;, err))<br> }<br><br> // Create clientset to interact with Kubernetes API<br> // Clientset provides access to all Kubernetes API groups and versions<br> // It&#39;s the main interface for performing CRUD operations on K8s resources<br> clientset, err := kubernetes.NewForConfig(config)<br> if err != nil {<br>  panic(fmt.Errorf(&quot;failed to create clientset: %v&quot;, err))<br> }<br><br> // Display successful connection information<br> fmt.Printf(&quot;Connected to external cluster: %s\n&quot;, config.Host)<br><br> // List all pods in the &quot;default&quot; namespace<br> // context.TODO() is used when context is needed but not available<br> podList, _ := clientset.CoreV1().Pods(&quot;default&quot;).List(context.TODO(), metav1.ListOptions{})<br><br> // Iterate through the list of pods and display their names<br> // podList.Items contains an array of Pod objects<br> for _, pod := range podList.Items {<br>  fmt.Printf(&quot;Pod Name: %s\n&quot;, pod.Name)<br> }<br>}</pre><h3>Create a Kubernetes Deployment Using Client-Go:</h3><p>Now let’s circle back to our initial question — creating deployments with <a href="https://github.com/kubernetes/client-go"><strong>client-go</strong></a> and understanding the interaction between <a href="https://github.com/kubernetes/client-go"><strong>client-go</strong></a> and the Kubernetes API. We’ll also see how to properly format our requests using the <a href="https://github.com/kubernetes/api"><strong>k8s.io/api</strong></a> and <a href="https://github.com/kubernetes/apimachinery"><strong>k8s.io/apimachinery</strong></a> modules before communicating with the Kubernetes API.</p><p>Following is the YAML format of the deployment that we intend to deploy programmatically. Let’s see how we can convert it to code:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-deployment<br>  namespace: default<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: my-app<br>  template:<br>    metadata:<br>      labels:<br>        app: my-app<br>    spec:<br>      containers:<br>      - name: app<br>        image: nginx:1.21</pre><p>Following is the code for creating a deployment using client-go.<br><em>( Note: Please check code comments for implementation details )</em></p><p>📂 <strong>GitHub:</strong> <a href="https://github.com/shamimice03/mastering-k8s-client-go/tree/main/02_deployment_using_client_go">deployment_using_client_go</a></p><pre>package main<br><br>import (<br> &quot;context&quot;<br> &quot;fmt&quot;<br> &quot;path/filepath&quot;<br><br> // k8s.io/api - Kubernetes resource definitions<br> appsv1 &quot;k8s.io/api/apps/v1&quot;<br> corev1 &quot;k8s.io/api/core/v1&quot;<br><br> // k8s.io/apimachinery - Common building blocks<br> metav1 &quot;k8s.io/apimachinery/pkg/apis/meta/v1&quot;<br><br> // k8s.io/client-go - Client library for Kubernetes API<br> &quot;k8s.io/client-go/kubernetes&quot;<br> &quot;k8s.io/client-go/rest&quot;<br> &quot;k8s.io/client-go/tools/clientcmd&quot;<br> &quot;k8s.io/client-go/util/homedir&quot;<br>)<br><br>// getExternalClusterConfig loads kubeconfig from ~/.kube/config<br>func getExternalClusterConfig() (*rest.Config, error) {<br> var kubeconfig string<br> if home := homedir.HomeDir(); home != &quot;&quot; {<br>  kubeconfig = filepath.Join(home, &quot;.kube&quot;, &quot;config&quot;)<br> }<br><br> // Build config from kubeconfig file<br> config, err := clientcmd.BuildConfigFromFlags(&quot;&quot;, kubeconfig)<br> if err != nil {<br>  return nil, fmt.Errorf(&quot;failed to build config from kubeconfig: %v&quot;, err)<br> }<br><br> return config, nil<br>}<br><br>// Helper function to convert int32 to *int32<br>func int32Ptr(i int32) *int32 {<br> return &amp;i<br>}<br><br>func main() {<br> // Get external cluster configuration<br> config, err := getExternalClusterConfig()<br> if err != nil {<br>  panic(fmt.Errorf(&quot;failed to get external cluster config: %v&quot;, err))<br> }<br><br> // Create clientset to interact with Kubernetes API<br> clientset, err := kubernetes.NewForConfig(config)<br> if err != nil {<br>  panic(fmt.Errorf(&quot;failed to create clientset: %v&quot;, err))<br> }<br><br> fmt.Printf(&quot;Connected to external cluster: %s\n&quot;, config.Host)<br><br> // Define a Deployment object<br> deployment := &amp;appsv1.Deployment{<br>  // TypeMeta - from apimachinery (API version and kind)<br>  TypeMeta: metav1.TypeMeta{<br>   APIVersion: &quot;apps/v1&quot;,<br>   Kind:       &quot;Deployment&quot;,<br>  },<br>  // ObjectMeta - from apimachinery (name, namespace, labels)<br>  ObjectMeta: metav1.ObjectMeta{<br>   Name:      &quot;nginx-deployment&quot;,<br>   Namespace: &quot;default&quot;,<br>  },<br>  // Spec - from k8s.io/api/apps/v1 (Deployment-specific configuration)<br>  Spec: appsv1.DeploymentSpec{<br>   Replicas: int32Ptr(3),<br>   Selector: &amp;metav1.LabelSelector{<br>    MatchLabels: map[string]string{<br>     &quot;app&quot;: &quot;nginx-app&quot;,<br>    },<br>   },<br>   // PodTemplateSpec - from k8s.io/api/core/v1<br>   Template: corev1.PodTemplateSpec{<br>    ObjectMeta: metav1.ObjectMeta{<br>     Labels: map[string]string{<br>      &quot;app&quot;: &quot;nginx-app&quot;,<br>     },<br>    },<br>    Spec: corev1.PodSpec{<br>     Containers: []corev1.Container{<br>      {<br>       Name:  &quot;nginx-app&quot;,<br>       Image: &quot;nginx:1.21&quot;,<br>      },<br>     },<br>    },<br>   },<br>  },<br> }<br><br> // Create Deployment using client-go<br> res, err := clientset.AppsV1().Deployments(&quot;default&quot;).Create(context.TODO(), deployment, metav1.CreateOptions{})<br><br> if err != nil {<br>  panic(fmt.Errorf(&quot;failed to create deployment: %v&quot;, err))<br> }<br><br> fmt.Printf(&quot;Successfully created deployment: %s\n&quot;, res.Name)<br>}</pre><h4>⏳ Codeflow:</h4><p>The following diagram shows the hierarchical structure of a Kubernetes Deployment object, breaking down from the main <strong>Deployment</strong> into <strong><em>TypeMeta</em></strong>, <strong><em>ObjectMeta</em></strong>, and <strong><em>Spec</em></strong> components, with each section containing specific fields like <strong><em>APIVersion</em></strong>, <strong><em>Name</em></strong>, <strong><em>Replicas</em></strong>, and <strong><em>Container</em></strong> details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tRgIdZGdLaDxhcVt_UT93Q.png" /><figcaption>Kubernetes Deployment Structure</figcaption></figure><p>With the following diagram, we can have a more in-depth view of how <a href="https://github.com/kubernetes/client-go"><strong>client-go</strong></a> is using structures from the <a href="https://github.com/kubernetes/api"><strong>k8s.io/api</strong></a> ( which are built on top of <a href="https://github.com/kubernetes/apimachinery/tree/master"><strong>API-Machinery</strong></a> foundations ) to interact with kubernetes resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ze9p6QufH0Bq3PR2UJiWww.png" /><figcaption>Inside client-go: Deployment resource architecture</figcaption></figure><p>In this blog, I tried to clarify how these three pieces work together, <a href="https://github.com/kubernetes/client-go"><strong>client-go</strong></a> serves as our programming toolkit, the API package defines what Kubernetes resources look like, and API-machinery provides the foundational framework that ties everything together.</p><blockquote><em>If you found this article useful, please click on the </em><strong><em>Follow </em></strong><em>👉 and </em><strong><em>Clap </em></strong><em>👏 buttons to support to write more about</em><strong><em> kubernetes client-go</em></strong><em>.<br>Thank you ❤</em></blockquote><h4>🔔 Follow me on <a href="https://www.linkedin.com/in/shamimice03/">LinkedIn </a>| <a href="http://github.com/shamimice03">GitHub </a>| <a href="https://twitter.com/shamimice03">Twitter</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dacda6fffef" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/getting-started-with-kubernetes-client-go-9dacda6fffef">Getting Started With Kubernetes Client-Go</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS IPAM: A Complete Guide to IP Address Management with AWS Organizations and RAM]]></title>
            <link>https://aws.plainenglish.io/aws-ipam-a-complete-guide-to-ip-address-management-with-aws-organizations-and-ram-7fc82d460b38?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fc82d460b38</guid>
            <category><![CDATA[aws-ram]]></category>
            <category><![CDATA[aws-ipam]]></category>
            <category><![CDATA[aws-vpc]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-organizations]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Mon, 20 Jan 2025 05:02:02 GMT</pubDate>
            <atom:updated>2025-06-03T16:21:27.626Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to effectively manage IP addresses across multiple AWS accounts and regions using AWS IPAM, Resource Access Manager (RAM), and Organizations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_wJyWti3K4Pz5fd12EjORA.png" /><figcaption>IPAM with AWS Organization and RAM</figcaption></figure><h4>Overview:</h4><p>Management of IP addresses can be challenging for medium and large-sized organizations. With multiple environments, accounts, and regions to handle, maintaining a consistent and organized IP address management strategy is crucial.</p><p>In this blog, we’ll explore how we can implement AWS IPAM (<a href="http://What is IPAM? - Amazon Virtual Private Cloud">IP Address Management</a>) in a structured way with <a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html">AWS Organizations</a> and <a href="https://aws.amazon.com/ram/">AWS Resource Access Manager</a>.</p><p>The following diagram shows the final structure after designing IPAM along with AWS Organizations and RAM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n9-21vbD4dAhaq9bU16osA.png" /><figcaption>IPAM with AWS Organization and RAM</figcaption></figure><h4>Prerequisites:</h4><ol><li>AWS Organization setup</li><li>A specific account for IPAM administration (<strong>AWS Organization Management Account will not work</strong>)</li></ol><p>The following regions and services will be used in this implementation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vFG2tcHo3KHkItAp1Ujvig.png" /></figure><h4>Structure of AWS Organization:</h4><p>The structure of AWS Organization is very important for this implementation because we will be sharing and allowing IPAM pools on the AWS Organization OU level. Of course, based on your needs you can always follow other approaches.</p><p>For this walkthrough, let’s use the following organizational structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m-sHi6bqKhti6FreKeb7VA.png" /><figcaption>AWS Organization Structure</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYUftig5Q7_tDk5MMij6sA.png" /><figcaption>AWS Organization Structure</figcaption></figure><p>We will use a dedicated account under the Mgmt OU as the IPAM administrator and host account.</p><h4>IPAM delegated administrator:</h4><p>To discover resources within an organization we have to delegate an account within the organization as an IPAM administrator from the organization management account. <br><strong>IMPORTANT:</strong> Please note that we cannot delegate the organization management account as the IPAM administrator.</p><p>We need to do this from our<strong> organization management account</strong>, designating the <strong>cloudterms-security</strong> account as our IPAM administrator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*vFT9Z3rr-kO5XzUIZgV7vg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BWJSOwXiZ2kIUNqeI_4Gbg.png" /><figcaption>IPAM delegated administrator</figcaption></figure><h4>AWS IPAM Structure:</h4><p>An IPAM can contain multiple scopes, with two default scopes provided: one for public IP management and another for private IP management. Our current focus is on the private scope. Within the private scope, we have to create a root pool, from which multiple sub-pools will be created. Then we need to distribute those sub-pools across various OUs within our organization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YkfDrsXJdi86Jrk_r9WUrw.png" /><figcaption>AWS IPAM Structure</figcaption></figure><h4>IPAM Creation Process:</h4><ul><li><strong>IPAM Tier:</strong> The advanced tier must be selected to manage private IPs within the organization. However, this tier incurs AWS costs. You can view the associated costs <a href="https://aws.amazon.com/vpc/pricing/"><strong>here</strong></a>.</li><li><strong>Operating Regions:</strong> Choosing operating regions based on your organization’s specific needs, such as the regions where workloads will be deployed, is crucial. Regions can be added at any time as requirements evolve.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MEAB0n29gFXonPncl9V1RQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*X4PPyZsL1uKnIdYLQhhXcw.png" /></figure><h4>Scopes:</h4><p>As mentioned earlier, once the IPAM is created, two scopes will be generated by default: one for private IP management and the other for public IP management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCGUeN1fXIh5Xn3UJ1i_uQ.png" /></figure><h4>Planning of IPAM Pools:</h4><p>Proper planning before creating pools is crucial. A well-thought-out plan can help avoid many future conflicts and hassles.</p><p>The structure outlined for this blog will be followed. Let’s go over each section in detail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gSkyLOGMkLlarB8CdGMs3A.png" /><figcaption>Planned AWS IPAM Structure</figcaption></figure><h4><strong>Root Pool: Foundation of IP Pools:</strong></h4><p>At the top level, we can begin with the Root Pool (10.0.0.0/8), offering an extensive IP address space of over 16 million addresses. This provides sufficient capacity for all current environments and future expansion. The <strong>Root Pool</strong> is configured with <strong>Locale: None</strong>as it serves solely as a container for all sub-pools.</p><blockquote><strong><em>What is locale?<br></em></strong><em>The locale refers to the AWS Region where the IPAM pool will be available for allocations. For instance, a CIDR block for a VPC can only be allocated from an IPAM pool that shares the same locale as the VPC’s Region. Once a locale is assigned to a pool, it cannot be changed.</em></blockquote><pre>Root_Pool:<br>- Scope: Private_Scope<br>- CIDR: 10.0.0.0/8<br>  - allocation_rule_settings:<br>    - Min: /12<br>    - Default: /20<br>    - Max: /24<br>- Locale: None</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UM_AT6dmI9vpmSjVgSEfTw.png" /></figure><h4>Environment-Level Pools:</h4><p>We can divide our <strong>Root Pool</strong> into several distinct /12 blocks, one for each environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ot7Ss0klnmEZitQPWsqusg.png" /></figure><p><strong>Dev Pool (10.0.0.0/12): </strong>This pool is designated for Development Accounts.</p><ul><li><strong>Locale:</strong> None, as the Dev Pool will serve as the parent pool for regional sub-pools.</li></ul><pre>Dev_Pool:<br>- Scope: Private_Scope<br>- Source: Root_Pool<br>- CIDR: 10.0.0.0/12<br>  - allocation_rule_settings:<br>    - Min: /14<br>    - Default: /20<br>    - Max: /24<br>- Locale: None</pre><p><strong>Prod Pool (10.16.0.0/12): </strong>This pool is reserved for Production Accounts.</p><ul><li><strong>Locale:</strong> None, as the Prod Pool will serve as the parent pool for regional sub-pools.</li></ul><pre>Pood_Pool:<br>- Scope: Private_Scope<br>- Source: Root_Pool<br>- CIDR: 10.16.0.0/12<br>  - allocation_rule_settings:<br>    - Min: /14<br>    - Default: /20<br>    - Max: /24<br>- Locale: None</pre><p><strong>Mgmt Pool (10.32.0.0/12): </strong>This pool is allocated for Management Accounts.</p><ul><li><strong>Locale:</strong> None, as the Mgmt Pool will serve as the parent pool for regional sub-pools.</li></ul><pre>Mgmt_Pool:<br>- Scope: Private_Scope<br>- Source: Root_Pool<br>- CIDR: 10.32.0.0/12<br>  - allocation_rule_settings:<br>    - Min: /14<br>    - Default: /20<br>    - Max: /24<br>- Locale: None</pre><h4>Regional Pool Allocation:</h4><p>Each environment pool is further divided into /14 blocks for specific regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCI1KQ2ig9aCzuNaMybhWg.png" /><figcaption>Regional Pool Allocation</figcaption></figure><p><strong>Development Regional Pools: Dev</strong></p><pre>Dev_Tokyo_Pool:<br>- Scope: Private_Scope<br>- Source: Dev_Pool<br>- CIDR: 10.0.0.0/14<br>  - allocation_rule_settings:<br>    - Min: /16<br>    - Default: /20<br>    - Max: /24<br>- Locale: ap-northeast-1<br><br>Dev_Osaka_Pool:<br>- Scope: Private_Scope<br>- Source: Dev_Pool<br>- CIDR: 10.4.0.0/14<br>  - allocation_rule_settings:<br>    - Min: /16<br>    - Default: /20<br>    - Max: /24<br>- Locale: ap-northeast-3<br><br>Dev_Virgina_Pool:<br>- Scope: Private_Scope<br>- Source: Dev_Poopyl<br>- CIDR: 10.8.0.0/14<br>  - allocation_rule_settings:<br>    - Min: /16<br>    - Default: /20<br>    - Max: /24<br>- Locale: us-east-1</pre><p>Similarly, we can configure for Prod and Mgmt regional pools:</p><pre>Production Regional Pools:<br>Tokyo (ap-northeast-1): 10.16.0.0/14<br>Osaka (ap-northeast-3): 10.20.0.0/14<br>N.Virginia (us-east-1): 10.24.0.0/14<br><br>Management Regional Pools:<br>Tokyo (ap-northeast-1): 10.32.0.0/14<br>Osaka (ap-northeast-3): 10.36.0.0/14<br>N.Virginia (us-east-1): 10.40.0.0/14</pre><h4>Sharing Regional Pools With OUs:</h4><p>The next step is to distribute these regional pools to the AWS Organization Organizational Units (OUs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PBmtsJEL6sU9Pt_HtoyOpg.png" /><figcaption>Sharing Regional Pools With OUs</figcaption></figure><p>Here are the steps to share IPAM pools using RAM (Resource Access Manager):</p><p><a href="https://docs.aws.amazon.com/vpc/latest/ipam/share-pool-ipam.html">Share an IPAM pool using AWS RAM</a></p><p><strong>**Important*** </strong>If you are using RAM for the first time, then you need to enable RAM across the AWS Organization from the AWS Organization management account:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-8ve4G0CJGFcskCwOsSPw.png" /></figure><p>The following settings must be configured from the IPAM Administrator account. All sharing should occur in the same region where IPAM is hosted. In this case, since IPAM is hosted in the <strong>ap-northeast-1</strong> region, all RAM shares need to be configured within the <strong>ap-northeast-1 </strong>region.</p><ul><li>Specify the resource to be shared; in our case, it is the regional pool:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yCYzbU5Mc6AOqVPkUG4DZw.png" /></figure><ul><li>Associate Managed Permissions:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kjJG8vxdZxcd-JijU9_aPw.png" /></figure><ul><li>Add OU as a principle:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ctdtxPMAAs7Dx95_vDyNOQ.png" /></figure><ul><li>And finally, share the resource:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*00KOjMtBMQwIOckcZCgK6Q.png" /></figure><ul><li>Following are some of the resource shares with Organizational Units (OUs):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6N1ngqDuA5Q5ykKkURoIXQ.png" /></figure><h4>Verification:</h4><p>Now, let’s switch to one of the accounts under the Dev OU. If we navigate to the RAM console for that account and check the <strong>Shared with Me</strong> section, we can view the shared resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S0jUJalDwC7QcDMcIzJk_w.png" /></figure><p><strong>Create a VPC with an IPAM Pool:</strong><br> Now, let’s proceed to create a VPC using an IPAM Pool. By selecting an <strong>IPAM-allocated IPv4 CIDR</strong> block, we can view the allowed IPAM pool. Using this allocation, we can create a new VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Usrp0t_niXq7zJtyWHR9pw.png" /></figure><h4>Enforce IPAM Pool Usage:</h4><p>To enforce the usage of IPAM pools across the organization, we can use the following SCP and attach it to the root or target OUs within the organization. This will ensure that the entire organization or target OUs utilizes IPAM pools when creating VPCs.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [{<br>       &quot;Effect&quot;: &quot;Deny&quot;,<br>        &quot;Action&quot;: [&quot;ec2:CreateVpc&quot;, &quot;ec2:AssociateVpcCidrBlock&quot;],<br>        &quot;Resource&quot;: &quot;arn:aws:ec2:*:*:vpc/*&quot;,<br>        &quot;Condition&quot;: {<br>            &quot;Null&quot;: {<br>                &quot;ec2:Ipv4IpamPoolId&quot;: &quot;true&quot;<br>            }<br>        }<br>     }]<br>}</pre><p>For more details, refer to the following resource:</p><p><a href="https://docs.aws.amazon.com/vpc/latest/ipam/scp-ipam.html">Enforce IPAM use for VPC creation with SCPs</a></p><h4>Monitoring:</h4><p>We can monitor IP usage, detect overlaps, and track various other metrics directly from the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2bp1k8tgV_knF0Typ30lyw.png" /><figcaption>Monitoring</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v0af6Zf-JLLzULl8.png" /></figure><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fc82d460b38" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/aws-ipam-a-complete-guide-to-ip-address-management-with-aws-organizations-and-ram-7fc82d460b38">AWS IPAM: A Complete Guide to IP Address Management with AWS Organizations and RAM</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing AWS Logs using AWS Athena (SQL)]]></title>
            <link>https://towardsaws.com/analyzing-aws-logs-using-aws-athena-sql-4188861cb774?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/4188861cb774</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws-elb]]></category>
            <category><![CDATA[aws-athena]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Tue, 01 Oct 2024 07:01:31 GMT</pubDate>
            <atom:updated>2025-06-03T16:19:08.445Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Comprehensive Hands-On on AWS Athena and ALB Access Logs Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rhz8-OADgEcqhJsy.png" /><figcaption>Image: <a href="https://aws.amazon.com/athena/">https://aws.amazon.com/athena/</a></figcaption></figure><h4>Introduction:</h4><p>This article will help you to get started with AWS Athena to analyze AWS logs. More specifically, this article will focus on ALB access logs and how to use SQL queries to analyze logs.</p><p>For the sake of brevity, let’s assume that we already have <strong>ALB</strong> running with <strong>access logs enabled</strong>.</p><ul><li>Check the <strong>Attributes -&gt; Monitoring</strong> to enable <strong>Access logs</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-f0w_DuR58VJl7UBXIaENw.png" /><figcaption>ALB Attributes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRMYI5NoHK6qaQytUZEVHA.png" /><figcaption>Access log storage location</figcaption></figure><ul><li>For details: <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html">Enable access logs for your Application Load Balancer.</a></li></ul><h4>AWS Athena Overview:</h4><p>AWS Athena enabled the option to analyze the unstructured, semi-structured, and structured data stored in Amazon S3 using simple SQL queries. In addition, query results can be saved and used for further reference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VgRogMBwayVgj3qnR4LDGQ.png" /><figcaption>Athena Console</figcaption></figure><p>By default, the data source will be <strong>AwsDataCatalog</strong>.Because Athena integrates with AWS Glue Data Catalog, which serves as a persistent metadata store for data stored in S3. This integration enables the creation of tables and the execution of queries in Athena using a centralized metadata store that is accessible throughout the entire AWS account.</p><p>For more details: <a href="https://docs.aws.amazon.com/athena/latest/ug/when-should-i-use-ate.html">Documentation</a></p><h4>Hands-On</h4><p>Let’s start the hands from here:</p><p><strong>Prerequisites:</strong></p><ul><li><strong>Query output location (*Required): </strong>We need to set a query output location in S3, this will allow us to store our query results on the specified bucket.</li><li><strong>Database creation (*Optional):</strong> If we want we can also create a database in AWS Athena for the logical grouping of tables we create.</li></ul><pre># To create database on athena<br><br>CREATE DATABASE elb_logs;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zTCCesVzbv2H-c3UA_Yw2Q.png" /><figcaption>Database Creation on Athena</figcaption></figure><p>For more details: <a href="https://docs.aws.amazon.com/athena/latest/ug/creating-databases.html">Documentation</a></p><ul><li><strong>Table Creation (*Required): </strong>From the Athena console, we can create a table. To create a table for ALB access logs we can follow the following procedure:</li></ul><p><a href="https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html">Query Application Load Balancer logs</a></p><h4>Create Table for ALB Logs:</h4><p>Since ALB access logs have a known structure, we can reduce query runtime and automate partition management using the Athena partition projection feature.</p><ul><li>Create a database table for ALB access logs with partition:</li></ul><pre>CREATE EXTERNAL TABLE IF NOT EXISTS alb_access_logs (<br>            type string,<br>            time string,<br>            elb string,<br>            client_ip string,<br>            client_port int,<br>            target_ip string,<br>            target_port int,<br>            request_processing_time double,<br>            target_processing_time double,<br>            response_processing_time double,<br>            elb_status_code int,<br>            target_status_code string,<br>            received_bytes bigint,<br>            sent_bytes bigint,<br>            request_verb string,<br>            request_url string,<br>            request_proto string,<br>            user_agent string,<br>            ssl_cipher string,<br>            ssl_protocol string,<br>            target_group_arn string,<br>            trace_id string,<br>            domain_name string,<br>            chosen_cert_arn string,<br>            matched_rule_priority string,<br>            request_creation_time string,<br>            actions_executed string,<br>            redirect_url string,<br>            lambda_error_reason string,<br>            target_port_list string,<br>            target_status_code_list string,<br>            classification string,<br>            classification_reason string,<br>            conn_trace_id string<br>            )<br>            PARTITIONED BY<br>            (<br>             day STRING<br>            )<br>            ROW FORMAT SERDE &#39;org.apache.hadoop.hive.serde2.RegexSerDe&#39;<br>            WITH SERDEPROPERTIES (<br>            &#39;serialization.format&#39; = &#39;1&#39;,<br>            &#39;input.regex&#39; = <br>        &#39;([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*)[:-]([0-9]*) ([-.0-9]*) ([-.0-9]*) ([-.0-9]*) (|[-0-9]*) (-|[-0-9]*) ([-0-9]*) ([-0-9]*) \&quot;([^ ]*) (.*) (- |[^ ]*)\&quot; \&quot;([^\&quot;]*)\&quot; ([A-Z0-9-_]+) ([A-Za-z0-9.-]*) ([^ ]*) \&quot;([^\&quot;]*)\&quot; \&quot;([^\&quot;]*)\&quot; \&quot;([^\&quot;]*)\&quot; ([-.0-9]*) ([^ ]*) \&quot;([^\&quot;]*)\&quot; \&quot;([^\&quot;]*)\&quot; \&quot;([^ ]*)\&quot; \&quot;([^\\s]+?)\&quot; \&quot;([^\\s]+)\&quot; \&quot;([^ ]*)\&quot; \&quot;([^ ]*)\&quot; ?([^ ]*)?&#39;<br>            )<br>            LOCATION &#39;s3://amzn-s3-demo-bucket/AWSLogs/&lt;ACCOUNT-NUMBER&gt;/elasticloadbalancing/&lt;REGION&gt;/&#39;<br>            TBLPROPERTIES<br>            (<br>             &quot;projection.enabled&quot; = &quot;true&quot;,<br>             &quot;projection.day.type&quot; = &quot;date&quot;,<br>             &quot;projection.day.range&quot; = &quot;2022/01/01,NOW&quot;,<br>             &quot;projection.day.format&quot; = &quot;yyyy/MM/dd&quot;,<br>             &quot;projection.day.interval&quot; = &quot;1&quot;,<br>             &quot;projection.day.interval.unit&quot; = &quot;DAYS&quot;,<br>             &quot;storage.location.template&quot; = &quot;s3://amzn-s3-demo-bucket/AWSLogs/&lt;ACCOUNT-NUMBER&gt;/elasticloadbalancing/&lt;REGION&gt;/${day}&quot;<br>            )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bsjkXF5pz6t5oWy_z3aeUA.png" /><figcaption>Table creation in athena</figcaption></figure><h4>Getting Started with Athena Query:</h4><p>We are all set up to run queries on ALB access logs stored in S3.</p><ul><li><strong>Query 1: To view all the attributes</strong></li></ul><pre>SELECT * FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot; limit 10;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZtGwVKJWY51svV4TpmDOQ.png" /></figure><ul><li><strong>Query 2: To view selected attributes</strong></li></ul><pre>SELECT type, time, client_ip, request_verb,request_url,elb_status_code <br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot; <br>LIMIT 100;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ohXqKY6n49CCsH_lrRxew.png" /></figure><ul><li><strong>Query 3: To view all the unique client IPs ( DISTINCT )</strong></li></ul><pre>SELECT DISTINCT client_ip <br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot; </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ctVc7Mtbs5cj3avzc_kWWA.png" /></figure><ul><li><strong>Query 4: Order client’s IP based on the status code (ORDER BY — Client IP)</strong></li></ul><pre>SELECT DISTINCT(client_ip), elb_status_code <br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot; <br>ORDER BY client_ip</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKytJpA2hqICiv2uQmI69Q.png" /></figure><ul><li><strong>Query 5: How many requests came from each IP?</strong></li></ul><pre>SELECT client_ip, COUNT(client_ip) AS number_of_requests<br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot;<br>GROUP BY client_ip<br>ORDER BY number_of_requests DESC;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r2IA4MFLXpbpSSU9EgV4JQ.png" /></figure><ul><li><strong>Query 6: How many requests came from each user-agent?</strong></li></ul><pre>SELECT COUNT(user_agent) AS request_count_from_agent ,user_agent<br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot;<br>GROUP BY user_agent<br>ORDER BY request_count_from_agent DESC;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-IiQlDRSf9fsUkr9qaP_g.png" /></figure><ul><li><strong>Query 7: Filter client IP based on status code:</strong></li></ul><pre>SELECT time, client_ip, request_verb, request_url, elb_status_code<br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot;<br>WHERE elb_status_code &gt;= 400;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtfEUK2CKZOhPgj47V-2eA.png" /></figure><ul><li><strong>Query 8: Filter client IP based on status code ( within a range ):</strong></li></ul><pre>SELECT day,client_ip, request_verb, request_url, elb_status_code<br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot;<br>WHERE day BETWEEN &#39;2024/08/16&#39; AND &#39;2024/08/17&#39;<br>ORDER BY day ASC;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MmPCRXg9XFLKpfw5mNgV_Q.png" /></figure><p>Similarly, we can use NOT BETWEEN to exclude a range</p><ul><li><strong>Query 9: Filter based on request verbs</strong></li></ul><pre>SELECT request_verb, request_url<br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot;<br>WHERE request_verb IN (&#39;GET&#39;, &#39;POST&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7Tu5sRjTZ7G0GP3Ky3BXA.png" /></figure><ul><li><strong>Query 10: Filter based on a pattern (using the LIKE operator)</strong></li></ul><p><strong>Example</strong>: Fetch the request URLs which has <strong>admin</strong> keyword.</p><pre>SELECT request_verb, request_url<br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot;<br>WHERE request_url LIKE &#39;%/admin/%&#39;; </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTU8-dXMGV7_-gjz5gxDQQ.png" /></figure><ul><li>NOT LIKE to exclude specific patterns.</li></ul><pre>SELECT request_verb, request_url<br>FROM &quot;elb_logs&quot;.&quot;alb_access_logs&quot;<br>WHERE request_url NOT LIKE &#39;%admin%&#39;; </pre><p><strong>More about pattern matching:</strong></p><p><a href="https://www.postgresql.org/docs/12/functions-matching.html">9.7. Pattern Matching</a></p><h4><strong>How to save quires in Athena?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1CH6eE7L-NjcEna6U50pw.png" /></figure><h4><strong>More about SQL aggregate functions:</strong></h4><p><a href="https://www.w3schools.com/sql/sql_aggregate_functions.asp">W3Schools.com</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N3hXOxxMgXyGZ8AB.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4188861cb774" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/analyzing-aws-logs-using-aws-athena-sql-4188861cb774">Analyzing AWS Logs using AWS Athena (SQL)</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS IAM Policy — ResourceTag VS RequestTag]]></title>
            <link>https://towardsaws.com/aws-iam-policy-resourcetag-vs-requesttag-16f19e4b83a3?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/16f19e4b83a3</guid>
            <category><![CDATA[aws-ami]]></category>
            <category><![CDATA[iam-policy]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-iam]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Tue, 10 Oct 2023 05:59:18 GMT</pubDate>
            <atom:updated>2023-10-10T05:59:18.400Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS IAM Policy — ResourceTag VS RequestTag</h3><h4>How to use ResourceTag and RequestTag inside the AWS Policy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmgNyp19MxeyiHns_WiWEg.png" /></figure><p>The goal of this article is to explain the difference between <strong>ResourceTag</strong> and <strong>RequestTag </strong>and their potential uses in IAM policies.</p><h4>ResourceTags:</h4><p><strong>Resource tags</strong> are the tags that are already attached to the resources. We can use <strong>aws:ResourceTag/key-name</strong> condition key to determine whether to allow access to the resources based on the tags attached. This is also known as attribute-based access control.</p><h4>RequestTags:</h4><p><strong>Request tags </strong>are tags that can be passed in an IAM request. Using <strong>aws:RequestTag/key-name</strong> condition key we can determine what tags (key-value pair) can be passed to tag AWS resources.</p><h3>Examples:</h3><h4><a href="#1">AllowEC2OnAllRegion</a>:</h4><ul><li>Let’s say we have created a new user, and then we want to grant certain permissions to that user. For example, if we want to allow the user to launch EC2 instances in all AWS regions.</li></ul><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {   <br>            &quot;Sid&quot;: &quot;ReadOnlyAccess&quot;,  <br>            &quot;Action&quot;: [<br>                &quot;ec2:Describe*&quot;,<br>                &quot;ec2:GetConsole*&quot;,<br>                &quot;cloudwatch:DescribeAlarms&quot;,<br>                &quot;cloudwatch:GetMetricStatistics&quot;,<br>                &quot;iam:ListInstanceProfiles&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;<br>        },<br>        {   <br>            &quot;Sid&quot;: &quot;AllowEC2OnAllRegion&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:RunInstances&quot;,<br>                &quot;ec2:CreateTags&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:ec2:*:*:instance/*&quot;,<br>                &quot;arn:aws:ec2:*:*:key-pair/*&quot;,<br>                &quot;arn:aws:ec2:*:*:security-group/*&quot;,<br>                &quot;arn:aws:ec2:*:*:volume/*&quot;,<br>                &quot;arn:aws:ec2:*:*:network-interface/*&quot;,<br>                &quot;arn:aws:ec2:*:*:subnet/*&quot;,<br>                &quot;arn:aws:ec2:*:*:image/ami-*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><ul><li>The “<strong>ReadOnlyAccess” </strong>sid will<strong> </strong>allow users/roles to view any EC2 resources using the EC2 console and also allows basic health information about EC2 using <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_DescribeAlarms.html">DescribeAlarms</a> and <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html">GetMetricStatistics</a> APIs. <a href="https://docs.aws.amazon.com/IAM/latest/APIReference/API_ListInstanceProfiles.html">ListInstanceProfiles</a> allows listing existing instance profiles on the instance launching page. But <a href="https://docs.aws.amazon.com/IAM/latest/APIReference/API_ListInstanceProfiles.html">ListInstanceProfiles</a> API does not allow the attachment of the IAM instance profile to EC2 instances.</li><li>The “<strong>AllowEC2OnAllRegion</strong>” sid will allow to provision of EC2 instances on all AWS regions. However, it also imposes limitations on users/roles from generating their own security groups and key pairs. To deploy EC2 instances under this policy, users/roles must utilize pre-existing security groups and key pairs.</li></ul><h4><a href="#2">OnlyAllowApprovedAMI:</a></h4><p>Suppose, while provisioning an EC2 instance, the user must use an <strong>approved AMI </strong>that has already been approved by specific teams. To achieve that AMI management team must attach certain tags to the approved AMIs. Users will be granted permission to use that AMI if the tag matches.</p><ul><li>AMI tagged with Approved=True</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*ZVYislz3CPNUfZ_89KBSgQ.png" /></figure><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {   <br>            &quot;Sid&quot;: &quot;ReadOnlyAccess&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:Describe*&quot;,<br>                &quot;ec2:GetConsole*&quot;,<br>                &quot;cloudwatch:DescribeAlarms&quot;,<br>                &quot;cloudwatch:GetMetricStatistics&quot;,<br>                &quot;iam:ListInstanceProfiles&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;<br>        },<br>        {   &quot;Sid&quot;: &quot;AllowEC2OnlyOnSpecificRegion&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:RunInstances&quot;,<br>                &quot;ec2:CreateTags&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:ec2:ap-northeast-1:*:instance/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:key-pair/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:security-group/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:volume/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:network-interface/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:subnet/*&quot;<br>            ]<br>        },<br>        {   &quot;Sid&quot;: &quot;OnlyApprovedAMI&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:RunInstances&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:ec2:ap-northeast-1::image/ami-*&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;ec2:ResourceTag/Approved&quot;: &quot;True&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><ul><li>The “<strong>AllowEC2OnlyOnSpecificRegion</strong>” sid will allow the provisioning of EC2 instances only on <strong>ap-northeast-1</strong> region.</li></ul><pre>        {   <br>            &quot;Sid&quot;: &quot;OnlyApprovedAMI&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:RunInstances&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:ec2:ap-northeast-1::image/ami-*&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;ec2:ResourceTag/Approved&quot;: &quot;True&quot;<br>                }<br>            }<br>        }</pre><ul><li>The “<strong>OnlyApprovedAMI</strong>” sid will allow the AMIs that are tagged with <strong>Approved = True</strong></li></ul><blockquote><strong><em>Use-case:</em></strong><em> In the above statement, </em><strong><em>ec2:ResourceTag</em></strong><em>/Approved:&quot;True” is used to filter existing AMIs. EC2 will be launched only if the specified tag is already attached to the resources.</em></blockquote><h4><a href="#3">EnforcingEC2Tagging:</a></h4><p>If we want to force the user to follow a resource tagging rule. For Example, users must tag the EC2 instance with <strong>Env: dev</strong> key-value pair before launching. Without attaching the specified tag EC2 instance won&#39;t be launched.</p><p><strong>The requirements listed below can be met with the following IAM policy:</strong></p><ul><li>EC2 should be launched on <strong>ap-northeast-1</strong> AWS region.</li><li>Only <strong>Approved = True</strong> tagged AMI can be used.</li><li>EC2 must be tagged with <strong>Env: dev</strong> key-pair while launching.</li></ul><p><strong>Following IAM policy will respect all our requirements mentioned above:</strong></p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {   <br>            &quot;Sid&quot;: &quot;ReadOnlyAccess&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:Describe*&quot;,<br>                &quot;ec2:GetConsole*&quot;,<br>                &quot;cloudwatch:DescribeAlarms&quot;,<br>                &quot;cloudwatch:GetMetricStatistics&quot;,<br>                &quot;iam:ListInstanceProfiles&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;<br>        },<br>        {   <br>            &quot;Sid&quot;: &quot;AllowEC2OnlyOnSpecificRegion&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;ec2:RunInstances&quot;,<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:ec2:ap-northeast-1:*:instance/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:key-pair/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:security-group/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:volume/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:network-interface/*&quot;,<br>                &quot;arn:aws:ec2:ap-northeast-1:*:subnet/*&quot;<br>            ]<br>        },<br>        {   <br>            &quot;Sid&quot;: &quot;EnforcingEC2Tagging&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;ec2:CreateTags&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;ec2:CreateAction&quot;: &quot;RunInstances&quot;,<br>                    &quot;aws:RequestTag/Env&quot;: &quot;dev&quot;<br>                }<br>            }<br>        },<br>        {  <br>            &quot;Sid&quot;: &quot;OnlyApprovedAMI&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:RunInstances&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:ec2:ap-northeast-1::image/ami-*&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;ec2:ResourceTag/Approved&quot;: &quot;True&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><p>This policy is similar to the previous one but there is an additional statement, that will make sure to attach a tag <strong>Env: dev</strong>before launching an instance on <strong>ap-northeast-1 </strong>region.</p><pre>{           <br>            &quot;Sid&quot;: &quot;EnforcingEC2Tagging&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;ec2:CreateTags&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;ec2:CreateAction&quot;: &quot;RunInstances&quot;,<br>                    &quot;aws:RequestTag/Env&quot;: &quot;dev&quot;<br>                }<br>            }<br>}</pre><p>As we can see we are using <strong>aws:RequestTag/Env: “dev”</strong>to force the user to add certain tags while passing the request to launch the EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*i6zraxRGhKIfmeygHBHHFQ.png" /></figure><ul><li>If we want to allow other values as tag values, then the policy statement will look like this:</li></ul><pre>        {   <br>            &quot;Sid&quot;: &quot;EnforcingEC2Tagging&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;ec2:CreateTags&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;ec2:CreateAction&quot;: &quot;RunInstances&quot;,<br>                    &quot;aws:RequestTag/Env&quot;: [<br>                        &quot;dev&quot;,<br>                        &quot;prod&quot;<br>                    ]<br>                }<br>            }<br>        }</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*QfCmFg7gMcccZIiEAnopZw.png" /></figure><blockquote><strong><em>Use-case:</em></strong><em> In the above policy, </em><strong><em>ec2:RequestTag/Env</em></strong><em>is used to force the user to pass tags </em><strong><em>Env: dev</em></strong><em>or </em><strong><em>Env: prod</em></strong><em>while launcing EC2 instance.</em></blockquote><h4>References:</h4><ul><li><a href="https://repost.aws/knowledge-center/restrict-launch-tagged-ami">Restrict access to launch EC2 instances from only tagged AMIs</a></li><li><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html#access_tags_control-resources">Controlling access to AWS resources using tags</a></li></ul><blockquote><em>If you found this article useful, please click on the </em><strong><em>Follow </em></strong><em>👉 and </em><strong><em>Clap </em></strong><em>👏 buttons to support me in writing more articles like this.<br>Thank you ❤</em></blockquote><h4>🔔 Follow me on <a href="https://www.linkedin.com/in/shamimice03/">LinkedIn </a>| <a href="http://github.com/shamimice03">GitHub </a>| <a href="https://twitter.com/shamimice03">Twitter</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16f19e4b83a3" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/aws-iam-policy-resourcetag-vs-requesttag-16f19e4b83a3">AWS IAM Policy — ResourceTag VS RequestTag</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Using Prometheus and Grafana]]></title>
            <link>https://levelup.gitconnected.com/monitoring-using-prometheus-and-grafana-f2bc85810ebb?source=rss-a96bc5a23088------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2bc85810ebb</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[server-monitoring]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[digitalocean]]></category>
            <dc:creator><![CDATA[Md Shamim]]></dc:creator>
            <pubDate>Fri, 29 Sep 2023 14:03:46 GMT</pubDate>
            <atom:updated>2023-09-29T14:03:46.448Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Comprehensive Guideline to Monitor Linux Servers Using Prometheus and Grafana</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*OdwQrCo2n17lVfw-WYl9Ig.png" /></figure><h4>Prometheus and Grafana:</h4><ul><li><strong>Prometheus </strong>is an open-source <strong>monitoring tool </strong>that stores time series data, such as metrics.</li><li><strong>Grafana </strong>enables <strong>visualization </strong>of data sourced from Prometheus or other data sources.</li></ul><p>In this article, we will explore how to use Prometheus and Grafana to monitor Linux servers. To perform the demonstration, <a href="https://medium.com/u/9ee8d41c2743">DigitalOcean</a> cloud platform will be used to spin up Prometheus, Grafana, and application servers.</p><h4>Server specifications:</h4><ul><li><strong>Operating System: </strong><strong>Rocky Linux 9</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZvp7ZKrUKTIxvRSkS_jcw.png" /></figure><blockquote><strong><em>Note:</em></strong><em> Before starting the installation process, it’s essential to check if </em><strong><em>SE Linux</em></strong><em> is </em><strong><em>enforced</em></strong><em> on the current system and </em><strong><em>disable it if enabled</em></strong><em>, as this article doesn’t cover </em><strong><em>SE Linux </em></strong><em>management during installation.</em></blockquote><blockquote><strong>To disable SE Linux — </strong><a href="https://github.com/shamimice03/monitoring-with-prometheus-grafana/blob/main/Installation/disable_selinux.sh"><strong>Script</strong></a></blockquote><h4>Step-1: Prometheus Server</h4><p>To install Prometheus Server and ensure that it runs as a Linux service, use following setup script:</p><pre>#!/bin/bash<br>set -x  # Enable debugging output<br><br># Update the package list and upgrade installed packages<br>sudo yum update -y<br><br># Upgrade the system to the latest available packages<br>sudo yum upgrade -y<br><br># Install wget<br>sudo yum install wget -y<br><br># Create a system group for Prometheus<br>sudo groupadd --system prometheus<br><br># Create a system user for Prometheus with /sbin/nologin shell<br>sudo useradd -s /sbin/nologin --system -g prometheus prometheus<br><br># Create necessary directories for Prometheus configuration and data<br>sudo mkdir -p /var/lib/prometheus<br>sudo mkdir -p /etc/prometheus/rules<br>sudo mkdir -p /etc/prometheus/rules.s<br>sudo mkdir -p /etc/prometheus/files_sd<br><br># Create a temporary directory for downloading and extracting Prometheus binaries<br>wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz -P /tmp<br>tar xvf &quot;/tmp/prometheus-2.46.0.linux-amd64.tar.gz&quot; --strip-components=1 -C /tmp<br><br># Move Prometheus binaries and configuration files to appropriate locations<br>sudo mv -f /tmp/prometheus /usr/local/bin<br>sudo mv -f /tmp/promtool /usr/local/bin<br>sudo mv -f /tmp/prometheus.yml /etc/prometheus/<br>sudo mv -f /tmp/consoles /etc/prometheus/<br>sudo mv -f /tmp/console_libraries /etc/prometheus/<br><br># Define the service unit file path<br>SERVICE_UNIT_FILE=&quot;/etc/systemd/system/prometheus.service&quot;<br><br># Check if the service unit file already exists<br>if [ -e &quot;$SERVICE_UNIT_FILE&quot; ]; then<br>    echo &quot;Service unit file $SERVICE_UNIT_FILE already exists.&quot;<br>    exit 1<br>fi<br><br># Use a here document to create the Prometheus service unit file<br>sudo tee &quot;$SERVICE_UNIT_FILE&quot; &gt; /dev/null &lt;&lt;EOF<br>[Unit]<br>Description=Prometheus<br>Documentation=https://prometheus.io/docs/introduction/overview/<br>Wants=network-online.target<br>After=network-online.target<br><br>[Service]<br>Type=simple<br>User=prometheus<br>Group=prometheus<br>ExecReload=/bin/kill -HUP \$MAINPID<br>ExecStart=/usr/local/bin/prometheus \\<br>  --config.file=/etc/prometheus/prometheus.yml \\<br>  --storage.tsdb.path=/var/lib/prometheus \\<br>  --web.console.templates=/etc/prometheus/consoles \\<br>  --web.console.libraries=/etc/prometheus/console_libraries \\<br>  --web.listen-address=0.0.0.0:9090 \\<br>  --web.external-url=<br>SyslogIdentifier=prometheus<br>Restart=always<br><br>[Install]<br>WantedBy=multi-user.target<br>EOF<br><br># Change ownership of Prometheus configuration and data directories<br>sudo chown -R prometheus:prometheus /etc/prometheus<br>sudo chown -R prometheus:prometheus /var/lib/prometheus<br><br># Set appropriate permissions for directories<br>sudo chmod -R 775 /etc/prometheus<br>sudo chmod -R 775 /var/lib/prometheus<br><br># Reload systemd to apply the new unit file<br>sudo systemctl daemon-reload<br><br># Start Prometheus service and enable it to start at boot<br>sudo systemctl start prometheus<br>sudo systemctl enable prometheus</pre><ul><li>Verify the installation by running the following command:</li></ul><pre>&gt;&gt; sudo systemctl status prometheus</pre><ul><li>The <strong>Prometheus </strong>server will be exposed on the port :<strong>9090 </strong>. To view Prometheus UI access <strong>&lt;prometheus-server-ip&gt;:9090</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vl0MSlqoEmB9wlFxKBv0Ew.png" /></figure><h4>Step 2: Application Servers and Node Exporters</h4><p>We will spin up two <strong>Application Servers/Droplets</strong> and then install <strong>Node Exporters</strong> on them so that the <strong>Prometheus</strong> server can <strong>scrape</strong> node metrics from the application servers.</p><p><strong>Node Exporter Installation Script:</strong></p><pre>#!/bin/bash<br>set -x  # Enable debugging output<br><br># Update the package list and upgrade installed packages<br>sudo yum update -y<br><br># Upgrade the system to the latest available packages<br>sudo yum upgrade -y<br><br># Install wget<br>sudo yum install wget -y <br><br># Create a system group and user for Prometheus with /sbin/nologin shell<br>sudo groupadd --system prometheus<br>sudo useradd -s /sbin/nologin --system -g prometheus prometheus<br><br># Create a directory to store Node Exporter binaries<br>sudo mkdir -p /var/lib/node<br><br># Download and extract the Node Exporter binaries from the Prometheus GitHub release<br>wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz -P /tmp<br>sudo tar xvf /tmp/node_exporter-1.6.1.linux-amd64.tar.gz  --strip-components=1 -C /tmp<br><br># Move Node Exporter binaries to appropriate locations<br>sudo mv -f /tmp/node_exporter /var/lib/node/<br><br># Define the service unit file path<br>SERVICE_UNIT_FILE=&quot;/etc/systemd/system/node.service&quot;<br><br># Check if the service unit file already exists<br>if [ -e &quot;$SERVICE_UNIT_FILE&quot; ]; then<br>    echo &quot;Service unit file $SERVICE_UNIT_FILE already exists.&quot;<br>    exit 1<br>fi<br><br># Create the Prometheus service unit file<br>sudo tee &quot;$SERVICE_UNIT_FILE&quot; &gt; /dev/null &lt;&lt;EOF<br>[Unit]<br>Description=Prometheus Node Exporter<br>Documentation=https://prometheus.io/docs/introduction/overview/<br>Wants=network-online.target<br>After=network-online.target<br><br>[Service]<br>Type=simple<br>User=prometheus<br>Group=prometheus<br>ExecReload=/bin/kill -HUP $MAINPID<br>ExecStart=/var/lib/node/node_exporter<br><br>SyslogIdentifier=prometheus_node_exporter<br>Restart=always<br><br>[Install]<br>WantedBy=multi-user.target<br>EOF<br><br># Change ownership and permissions of Prometheus configuration and data directories<br>sudo chown -R prometheus:prometheus /var/lib/node<br>sudo chmod -R 775 /var/lib/node<br><br># Reload systemd to apply the new unit file<br>sudo systemctl daemon-reload<br><br># Start Prometheus service and enable it to start at boot<br>sudo systemctl start node<br>sudo systemctl enable node</pre><ul><li>Verify the installation of the <strong>node exporter</strong> using the following command:</li></ul><pre>&gt;&gt; sudo systemctl status node</pre><ul><li>Node Exporters will be exposed on port:9100 . To view metrics of the nodes: <strong>&lt;node-ip&gt;:9100/metrics</strong></li></ul><h4>Step 3: Service Discovery</h4><p>In this demonstration, we will use the <strong>service discovery </strong>method to scrape metrics from application servers. To achieve that in <a href="https://medium.com/u/9ee8d41c2743">DigitalOcean</a><strong>,</strong> we have to use <strong>digitalocean_sd_configs</strong> to scrape metrics.</p><p><strong>Prerequisites:</strong></p><ul><li>We need to add <strong>tags </strong>to the application servers/droplets to allow easy filtering of desired servers/droplets from a larger pool of servers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yUU5rqhLzJJrCK9r354TAw.png" /></figure><ul><li>Also <a href="https://medium.com/u/9ee8d41c2743"><strong>DigitalOcean</strong></a><strong> API token</strong> is required to discover servers/droplets running in the <a href="https://medium.com/u/9ee8d41c2743"><strong>DigitalOcean</strong></a><strong> </strong>platform</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rSAbEhMuXSFmYuRJFuwYNg.png" /></figure><p>Now, we are all set to configure <strong>service discovery.</strong> Add <strong>service discovery</strong> configurations in the <strong>/etc/prometheus/prometheus.yaml</strong> file:</p><pre># /etc/prometheus/prometheus.yaml<br><br># omitted ..<br><br><br># A scrape configuration containing exactly one endpoint to scrape:<br># Here it&#39;s Prometheus itself.<br>scrape_configs:<br>  - job_name: &quot;prometheus&quot;<br>    static_configs:<br>      - targets: [&quot;localhost:9090&quot;]<br><br>  # service discovery config<br>  - job_name: &quot;node&quot;<br>    digitalocean_sd_configs: <br>    - authorization:<br>          credentials: &quot;&lt;digital-ocean-api-token&quot;<br>      port: 9100<br>      refresh_interval: 60s<br>    relabel_configs:<br>      # Only scrape targets that have a tag &#39;application-server&#39; and &#39;webapp&#39;.<br>      - source_labels: [__meta_digitalocean_tags]<br>        regex: &quot;.*application-server.*webapp.*&quot;<br>        action: keep<br>      # Adding new label named `identifier`<br>      - source_labels: [__meta_digitalocean_droplet_name, __meta_digitalocean_private_ipv4]<br>        separator: &quot;@&quot;<br>        regex: &quot;(.*)@(.*)&quot;<br>        replacement: &quot;${1}@${2}&quot;<br>        target_label: identifier</pre><blockquote>Complete configuration file — <a href="https://github.com/shamimice03/monitoring-with-prometheus-grafana/blob/main/Prometheus-server/prometheus.yaml"><strong>prometheus.yaml</strong></a></blockquote><p>After making changes to the configuration file we can validate the configurations using <strong>promtool </strong>utility, which is installed as part of the prometheus installation process.</p><pre>promtool check config /etc/prometheus/prometheus.yml</pre><blockquote>In the configuration file, <strong>relable_configs</strong>is used to create new target label and filter droplets/servers among all servers. Explore more about <strong>Relabel Configs </strong>from here — <a href="https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/"><strong>Relable Configs</strong></a></blockquote><ul><li>Restart the <strong>Prometheus </strong>service and check the <strong>status of the service discovery </strong>using the Prometheus console UI.</li></ul><pre>&gt;&gt; sudo systemctl restart prometheus</pre><ul><li>View the <strong>Status</strong> of the Service Discovery:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVErgJ9S32iysERKD1jbiQ.png" /></figure><ul><li>Job named <strong>node</strong> is detected:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*2rr5bIVpoA9jL79q3tKtAw.png" /></figure><ul><li>New <strong>Target Labels(identifier)</strong> is created :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7xyR0zVqa4XakTGAKVlCw.png" /></figure><ul><li>Any droplets/servers that does not match <strong>tags </strong>defined in the <strong>relabel_configs </strong>section will be dropped.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*smFknslGf5vu-bsbLkoaTw.png" /></figure><blockquote><strong>Note: </strong>In future, any server/droplet provisioned with tag value<strong><em>application-server</em></strong> and <strong>webapp</strong> will be discoverd by the<strong> service descovery</strong> and kept to <strong>scrape </strong>metrics. That is the beauty of <strong>service discovery</strong> in prometheus.</blockquote><h4>Step 4: Metrics and Query</h4><p>Now, we should be able to view metrics from Prometheus Console UI and perform queries. Let’s run a query to fetch the<strong> free disk space (in bytes)</strong> of the root filesystem (‘/’) on the<strong> application servers.</strong></p><pre>node_filesystem_free_bytes{mountpoint=&quot;/&quot;}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7lY_wd70lsSTeKZURMxMyQ.png" /></figure><h4>Step-5: Grafana — Data Visualization</h4><p>We can utilize <strong>Grafana </strong>in order to effectively display the data collected by Prometheus.</p><p><strong>Script to install Grafana:</strong></p><pre>#!/bin/bash<br>set -x  # Enable debugging output<br><br>sudo yum update -y<br>sudo yum install -y https://dl.grafana.com/enterprise/release/grafana-enterprise-10.0.0-1.x86_64.rpm<br>sudo systemctl start grafana-server<br>sudo systemctl enable grafana-server<br>sudo systemctl status grafana-server</pre><h4>◎Add Prometheus as a Data Source:</h4><p>The Grafana server will be exposed on port :3000 . The initial username/password of Grafana is admin/admin . During the first login into the grafana server new password should be created.</p><ul><li>Access Grafana UI : &lt;node-ip&gt;:3000</li><li>Add<strong> Prometheus </strong>as a <strong>Data Source:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*KbA8I9cSqDACzeIsPPTi0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*cC0twG-e7BcH27fhePo1kg.png" /></figure><ul><li>Add Prometheus server URL:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*swE_f1IyYXA8W0wTIiIO3Q.png" /></figure><ul><li>It is also possible to import some default dashboards to monitor Prometheus and Grafana server itself:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*RLsd4dX3ahGEKRFUFyM3XQ.png" /></figure><ul><li>Then save and test the settings:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*ZkDAzvF9yvYVXU6iYatGEw.png" /></figure><h4>◎Dashboard to Monitor Hosts</h4><p>We can easily create a dashboard using Grafana UI. But for now, we will be using open sourced dashboard created by third parties.</p><ul><li>Discover numerous open-source Grafana dashboards at the following link:</li></ul><p><a href="https://grafana.com/grafana/dashboards/">Grafana dashboards | Grafana Labs</a></p><ul><li>We will be using <strong>Node Exporter Full</strong> named dashboard to monitor host metrics:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*bKh7XnKJrVj7-HTZ_-M7tg.png" /></figure><ul><li>Copy the <strong>Template Id</strong>to the clipboard:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5VXf_DbGbNTD857_y3plA.png" /></figure><ul><li>Move to the <strong>Dashboards </strong>option and <strong>import </strong>a new dashboard using the <strong>Template Id </strong>copied earlier.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*nGVTAWUtBhMjdO-1XhTlUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*oXaOiQ0mBLeQ5vauWgolsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qS1uXTKDTXAzGF0EhGcokg.png" /></figure><ul><li>Following is the dashboard we are looking for. Explore this dashboard to monitor hosts/droplets.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MyNE93Dtv5_JRaLs9OjCeg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pGhKaEI4QFHJ2SBJJSKUng.png" /></figure><h4><strong>Github Repository related to this article:</strong></h4><p><a href="https://github.com/shamimice03/monitoring-with-prometheus-grafana">GitHub - shamimice03/monitoring-with-prometheus-grafana: Monitoring Using Prometheus and Grafana</a></p><p>In the upcoming article, we will explore how to configure different rules (recording rules and alerting rules) in the <a href="https://medium.com/u/19f6b7f36d21">Prometheus Monitoring</a> server and generate alerts for anomalies.</p><blockquote><em>If you found this article useful, please click on the </em><strong><em>Follow </em></strong><em>👉 and </em><strong><em>Clap </em></strong><em>👏 buttons to support me in writing more articles like this.<br>Thank you ❤</em></blockquote><h4>🔔 Follow me on <a href="https://www.linkedin.com/in/shamimice03/">LinkedIn </a>| <a href="http://github.com/shamimice03">GitHub </a>| <a href="https://twitter.com/shamimice03">Twitter</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2bc85810ebb" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/monitoring-using-prometheus-and-grafana-f2bc85810ebb">Monitoring Using Prometheus and Grafana</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>