<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pyae Phyoe Shein on Medium]]></title>
        <description><![CDATA[Stories by Pyae Phyoe Shein on Medium]]></description>
        <link>https://medium.com/@pyaephyoeshein?source=rss-bb9ba485dc06------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_E8zsKRy3OUTCIfDvZ7a6A.jpeg</url>
            <title>Stories by Pyae Phyoe Shein on Medium</title>
            <link>https://medium.com/@pyaephyoeshein?source=rss-bb9ba485dc06------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:04:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pyaephyoeshein/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond the Waterline: How “The Great Flood” Teaches AI Humanity]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pyaephyoeshein.medium.com/beyond-the-waterline-how-the-great-flood-teaches-ai-humanity-e03b0af34965?source=rss-bb9ba485dc06------2"><img src="https://cdn-images-1.medium.com/max/1120/1*OEo65jevRBMfTtreP7caJg.png" width="1120"></a></p><p class="medium-feed-snippet">Honestly, I am not a big fan of Korean movies. Sometimes I watch a few comedies or dramas with my wife when we have some spare time. But&#x2026;</p><p class="medium-feed-link"><a href="https://pyaephyoeshein.medium.com/beyond-the-waterline-how-the-great-flood-teaches-ai-humanity-e03b0af34965?source=rss-bb9ba485dc06------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://pyaephyoeshein.medium.com/beyond-the-waterline-how-the-great-flood-teaches-ai-humanity-e03b0af34965?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/e03b0af34965</guid>
            <category><![CDATA[the-great-flood]]></category>
            <category><![CDATA[korean-dramas]]></category>
            <category><![CDATA[movie-review]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Tue, 23 Dec 2025 20:35:48 GMT</pubDate>
            <atom:updated>2026-01-14T09:49:10.306Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How I Built a Kubernetes Deployment Template for my DevOps Interview]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pyaephyoeshein.medium.com/how-i-built-a-kubernetes-deployment-template-for-my-devops-interview-7fb16bb58ce7?source=rss-bb9ba485dc06------2"><img src="https://cdn-images-1.medium.com/max/1280/1*wFjgg3f_9ofcuvTf5nPyuQ.png" width="1280"></a></p><p class="medium-feed-snippet">In today&#x2019;s cloud-native world, deploying applications reliably to Kubernetes environments&#xA0;, across different teams, services and stages&#xA0;&#x2026;</p><p class="medium-feed-link"><a href="https://pyaephyoeshein.medium.com/how-i-built-a-kubernetes-deployment-template-for-my-devops-interview-7fb16bb58ce7?source=rss-bb9ba485dc06------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://pyaephyoeshein.medium.com/how-i-built-a-kubernetes-deployment-template-for-my-devops-interview-7fb16bb58ce7?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fb16bb58ce7</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[jenkins-pipeline]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Wed, 05 Nov 2025 18:48:25 GMT</pubDate>
            <atom:updated>2025-11-05T18:59:30.695Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Migrating Node.js]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pyaephyoeshein.medium.com/migrating-node-js-b644f6359354?source=rss-bb9ba485dc06------2"><img src="https://cdn-images-1.medium.com/max/1920/1*OxjMN7xC6zu35KBpdjq3mw.png" width="1920"></a></p><p class="medium-feed-snippet">Roughly seven years ago, while I was working in Myanmar (Burma) on an outsourced project, our backend stack was built on Node.js with a&#x2026;</p><p class="medium-feed-link"><a href="https://pyaephyoeshein.medium.com/migrating-node-js-b644f6359354?source=rss-bb9ba485dc06------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://pyaephyoeshein.medium.com/migrating-node-js-b644f6359354?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/b644f6359354</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[elastic-beanstalk]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Sat, 11 Oct 2025 17:15:38 GMT</pubDate>
            <atom:updated>2025-10-11T17:38:58.343Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Cloud-Native Infrastructure: My Master’s Capstone Journey]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pyaephyoeshein.medium.com/building-a-cloud-native-infrastructure-my-masters-capstone-journey-31d6288b13b6?source=rss-bb9ba485dc06------2"><img src="https://cdn-images-1.medium.com/max/1200/1*8GQGrMI09h-EaoTZnhwTWw.png" width="1200"></a></p><p class="medium-feed-snippet">When I started my master&#x2019;s program, I knew the capstone project would be the ultimate test of everything I had learned. After months of&#x2026;</p><p class="medium-feed-link"><a href="https://pyaephyoeshein.medium.com/building-a-cloud-native-infrastructure-my-masters-capstone-journey-31d6288b13b6?source=rss-bb9ba485dc06------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://pyaephyoeshein.medium.com/building-a-cloud-native-infrastructure-my-masters-capstone-journey-31d6288b13b6?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/31d6288b13b6</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[helm-chart]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Sun, 05 Oct 2025 11:57:47 GMT</pubDate>
            <atom:updated>2025-10-05T18:03:33.723Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Automating NGINX Reverse Proxy with SSL Using Ansible Roles]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pyaephyoeshein.medium.com/automating-nginx-reverse-proxy-with-ssl-using-ansible-roles-1052c5a5bf33?source=rss-bb9ba485dc06------2"><img src="https://cdn-images-1.medium.com/max/2001/1*Ur4AQ8OHRogGni8SgH_rmQ.png" width="2001"></a></p><p class="medium-feed-snippet">Managing servers manually can be time-consuming and error-prone, especially when you are running multiple applications in containers&#x2026;</p><p class="medium-feed-link"><a href="https://pyaephyoeshein.medium.com/automating-nginx-reverse-proxy-with-ssl-using-ansible-roles-1052c5a5bf33?source=rss-bb9ba485dc06------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://pyaephyoeshein.medium.com/automating-nginx-reverse-proxy-with-ssl-using-ansible-roles-1052c5a5bf33?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/1052c5a5bf33</guid>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[letencrypt]]></category>
            <category><![CDATA[certbot]]></category>
            <category><![CDATA[ansible-playbook]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 19:09:05 GMT</pubDate>
            <atom:updated>2025-09-26T19:09:05.382Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Automating CI/CD with Jenkins DSL Plugins on Kubernetes Using Helm]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pyaephyoeshein.medium.com/automating-ci-cd-with-jenkins-dsl-plugins-on-kubernetes-using-helm-d497c606b9af?source=rss-bb9ba485dc06------2"><img src="https://cdn-images-1.medium.com/max/750/1*Nb43HUjZbkp6zZgKkJN8Wg.jpeg" width="750"></a></p><p class="medium-feed-snippet">In today&#x2019;s cloud-native world, DevOps teams are constantly seeking ways to streamline CI/CD pipelines. Jenkins remains a powerful tool in&#x2026;</p><p class="medium-feed-link"><a href="https://pyaephyoeshein.medium.com/automating-ci-cd-with-jenkins-dsl-plugins-on-kubernetes-using-helm-d497c606b9af?source=rss-bb9ba485dc06------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://pyaephyoeshein.medium.com/automating-ci-cd-with-jenkins-dsl-plugins-on-kubernetes-using-helm-d497c606b9af?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/d497c606b9af</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[jenkins-plugins]]></category>
            <category><![CDATA[jenkins-pipeline]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Wed, 27 Aug 2025 09:42:49 GMT</pubDate>
            <atom:updated>2025-08-27T09:42:49.653Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Automating Logstash Setup with Ansible: From Zero to Centralized Logging]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.datadriveninvestor.com/automating-logstash-setup-with-ansible-from-zero-to-centralized-logging-d08245098176?source=rss-bb9ba485dc06------2"><img src="https://cdn-images-1.medium.com/max/1536/1*9d5rkNuzJhs7NBhEkoeoRg.png" width="1536"></a></p><p class="medium-feed-snippet">&#x1F9ED; Introduction</p><p class="medium-feed-link"><a href="https://medium.datadriveninvestor.com/automating-logstash-setup-with-ansible-from-zero-to-centralized-logging-d08245098176?source=rss-bb9ba485dc06------2">Continue reading on DataDrivenInvestor »</a></p></div>]]></description>
            <link>https://medium.datadriveninvestor.com/automating-logstash-setup-with-ansible-from-zero-to-centralized-logging-d08245098176?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/d08245098176</guid>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[magento]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Sun, 03 Aug 2025 19:24:00 GMT</pubDate>
            <atom:updated>2025-08-18T10:46:00.940Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Centralized Kubernetes Log Streaming with EFKK stacks in Kubernetes]]></title>
            <link>https://pyaephyoeshein.medium.com/centralized-kubernetes-log-streaming-with-efkk-stacks-in-kubernetes-2a2ac60976f6?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a2ac60976f6</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[centralized-logging]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[efk]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Sun, 29 Jun 2025 18:25:31 GMT</pubDate>
            <atom:updated>2025-06-29T18:25:31.092Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KAUxXtx0hzCPQTNk_05RoQ.png" /></figure><p>When I started my first observability-focused role at a company in Thailand, my responsibilities revolved around improving our systems for monitoring, logging, and alerting. It was a great learning opportunity, especially considering the environment: the organization was structured into multiple units, each managing several microservices in Kubernetes.</p><p>This setup presented a challenge — and an opportunity. With dozens of services running across multiple teams, designing a centralized log streaming solution became a key priority for enabling transparency, reducing operational overhead, and improving troubleshooting across environments.</p><p>In this post, I’ll walk through the architecture I helped implement and learned from: a Kubernetes-native logging stack using Elasticsearch, Fluentd, Kibana (EFK), AWS Kinesis, and DaemonSets, built with Helm templating for multi-team flexibility.</p><h3>🌐 Why Centralized Logging Matters</h3><p>When working in a platform or DevOps capacity, it’s common to support multiple applications or services managed by different teams. Without a central logging system, observability becomes fragmented — logs are scattered, access is inconsistent, and teams waste time chasing down issues.</p><p>We needed a way to:</p><ol><li>Aggregate logs across all services and namespaces</li><li>Retain the ability to route logs by team or function</li><li>Securely ship logs to cloud-native services</li><li>Allow easy access to visualizations and queries</li></ol><h3>🧱 Stack Overview</h3><p>Here’s the core architecture we used to address these needs:</p><ol><li>Fluentd (via DaemonSet): Collect logs directly from Kubernetes nodes.</li><li>Kinesis Streams: Handle log forwarding to cloud storage or analytics platforms.</li><li>Elasticsearch: Store logs with indexing for fast search.</li><li>Kibana: Provide visual log dashboards.</li><li>Helm: Manage configurations and enable per-team customizations.</li><li>IAM + ServiceAccount: Secure access to cloud resources like Kinesis.</li></ol><h3>🛠️ Fluentd as a DaemonSet: Collecting Logs at Node Level</h3><p>We deployed Fluentd as a DaemonSet, which ensures a pod runs on every node in the Kubernetes cluster. This approach allows Fluentd to tail log files under <strong>`/var/log/containers/`</strong>, minimizing latency and avoiding the complexity of sidecar containers.</p><p>Here’s a simplified DaemonSet structure:</p><pre>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: fluentd<br>spec:<br>  template:<br>    spec:<br>      serviceAccountName: fluentd-serviceaccount<br>      containers:<br>      - name: fluentd<br>        image: fluent/fluentd-kubernetes-daemonset:v1.14<br>        volumeMounts:<br>        - name: varlog<br>          mountPath: /var/log<br>      volumes:<br>      - name: varlog<br>        hostPath:<br>          path: /var/log</pre><p>This configuration allows Fluentd to ingest logs from all containers running on a node and apply filters or enrich them with metadata before forwarding.</p><h3>🔐 Secure Access with IAM and ServiceAccounts</h3><p>Fluentd needs permission to send logs to external systems — such as AWS Kinesis. Instead of hardcoding credentials, we used IAM roles bound to Kubernetes ServiceAccounts (via IRSA on EKS). This ensures that each logging agent only has access to the specific resources it needs.</p><p>A ServiceAccount configuration with annotation might look like:</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: fluentd-serviceaccount<br>  namespace: logging<br>  annotations:<br>    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fluentd-logs-role</pre><p>In a Helm chart, this can be dynamically generated per team or environment.</p><h3>🔄 Fluentd Output: Kinesis + Elasticsearch</h3><p>We configured Fluentd to send logs to both AWS Kinesis Data Streams and an Elasticsearch cluster. Kinesis acts as a scalable buffer or stream processor (especially useful if connecting to Lambda or Firehose), while Elasticsearch supports fast querying through Kibana.</p><pre>&lt;match **&gt;<br>  @type copy<br>  &lt;store&gt;<br>    @type kinesis_streams<br>    stream_name &quot;my-k8s-logs&quot;<br>    region &quot;ap-southeast-1&quot;<br>    data_type &quot;json&quot;<br>  &lt;/store&gt;<br>  &lt;store&gt;<br>    @type elasticsearch<br>    host &quot;elasticsearch.logging.svc.cluster.local&quot;<br>    port 9200<br>    logstash_format true<br>  &lt;/store&gt;<br>&lt;/match&gt;</pre><p>This “dual stream” setup provides both operational visibility and long-term analytical capability.</p><h3>📊 Visualizing with Kibana</h3><p>Once logs are indexed in Elasticsearch, teams can use Kibana dashboards to monitor service behavior, trace issues, or even track deployments. Index patterns were structured in a way that teams could view their own logs without seeing others’ — a simple but effective form of multi-tenancy.</p><p>Dashboards were often customized by:</p><ol><li>Namespace</li><li>Service label</li><li>Log severity</li><li>Deployment stage (dev/staging/prod)</li></ol><h3>🔁 Helm Templates for Multi-Team Deployments</h3><p>To simplify deployment and encourage reuse, we wrapped the Fluentd DaemonSet and associated resources into a Helm chart, making it easy to deploy consistent logging agents across namespaces or teams.</p><p>We also included logic in the templates to handle per-team customization. For example, the following Helm helper lets us dynamically assign IAM roles based on a team’s config:</p><pre>{{- define &quot;logging.serviceAccountRole&quot; -}}<br>{{- $team := get .Values .Values.team  -}}<br>{{ default &quot;default&quot; $team.serviceAccount.iamrole }}<br>{{- end }}</pre><p>This approach reduced duplication and made it easy to onboard new services or teams with minimal setup.</p><h3>⚡ Key Takeaways</h3><ol><li>DaemonSets are effective for node-level log collection in Kubernetes.</li><li>Combining Fluentd + Kinesis + Elasticsearch balances scalability with observability.</li><li>ServiceAccount + IAM provides secure, auditable access to cloud services.</li><li>Helm templating is essential when managing multiple teams or business units.</li><li>Keeping logs structured and tagged (e.g., with team, environment, severity) pays off in the long run.</li></ol><h3>👣 Final Thoughts</h3><p>While every organization’s needs are different, these patterns and tools are widely applicable for teams running Kubernetes at scale. Observability is more than a dashboard — it’s a foundation for operational excellence.</p><p>Whether you’re just starting to build your logging pipeline or optimizing an existing one, I hope these insights help you think about structure, security, and scalability in your own environment.</p><p>Thanks for reading! If you found this post helpful, feel free to leave a comment or connect. Happy logging! 📘📊🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a2ac60976f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From ECS to EC2: How I Slashed AWS Costs for Non-Production Environments with CodePipeline and…]]></title>
            <link>https://awstip.com/from-ecs-to-ec2-how-i-slashed-aws-costs-for-non-production-environments-with-codepipeline-and-fa3ffff40572?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa3ffff40572</guid>
            <category><![CDATA[cloudformation]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws-ecs-fargate]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Mon, 05 May 2025 20:04:29 GMT</pubDate>
            <atom:updated>2025-05-15T12:02:08.550Z</atom:updated>
            <content:encoded><![CDATA[<h3>From ECS to EC2: How I Slashed AWS Costs for Non-Production Environments with CodePipeline and Docker</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovRuAuqPf4r2xpiWh71rUg.png" /><figcaption>image: <a href="https://logz.io/blog/what-is-docker/">https://logz.io/blog/what-is-docker/</a></figcaption></figure><p>When you’re building scalable microservices, Amazon ECS is an attractive option. It’s robust, integrates well with AWS services, and provides native support for Docker. But like many others, I faced an all-too-common issue: <strong>skyrocketing AWS bills</strong>, especially in lower environments where full ECS capabilities are often overkill.</p><h3>💸 The Problem: ECS is Great — Until the Bill Arrives</h3><p>In our architecture, each microservice is containerized and deployed via ECS. This setup worked flawlessly for production, but for <strong>non-production environments (DEV, QA, STAGING)</strong>, we started seeing significant costs:</p><ul><li>Each ECS service required its own <strong>ALB Target Group</strong>, and even though we reused the <strong>same Load Balancer</strong>, the cost of listeners and target groups added up.</li><li>Idle containers and unused capacity in ECS clusters were still costing money.</li></ul><p>So I asked myself:</p><blockquote>Do non-prod environments really need a high-availability, scalable ECS setup?</blockquote><h3>🧠 The Solution: Consolidate to EC2 + Docker + NGINX</h3><p>The idea was simple but effective:</p><blockquote><em>Use a single EC2 instance to host multiple services as Docker containers, and configure NGINX as a reverse proxy to route traffic.</em></blockquote><p>This approach provided:</p><ul><li><strong>One EC2 instance for all non-prod apps</strong></li><li><strong>Docker containers for service isolation</strong></li><li><strong>NGINX proxy</strong> for routing requests to each app</li><li><strong>CodePipeline + CodeBuild</strong> to automate Docker image builds and SSH-based deployments</li></ul><p>This move <strong>cut our AWS costs significantly</strong> while maintaining full CI/CD capability.</p><h3>🛠 Stack Overview</h3><p>Here’s a breakdown of the core AWS components:</p><ul><li><strong>AWS CodeCommit</strong>: Git repository</li><li><strong>AWS CodePipeline</strong>: CI/CD pipeline orchestrator</li><li><strong>AWS CodeBuild</strong>: Docker builder + deploy executor</li><li><strong>Amazon S3</strong>: Build artifact storage</li><li><strong>AWS Secrets Manager</strong>: For storing PEM keys and .env variables</li><li><strong>Amazon EC2</strong>: Running all containers behind a single NGINX proxy</li></ul><h3>⚙️ NGINX as a Smart Traffic Router</h3><p>Each microservice is proxied by NGINX like this:</p><pre>location /my-service/ {<br>    proxy_pass http://localhost:1339/;<br>}</pre><h3>🔁 Build and Deploy Flow</h3><p>With CodePipeline and CodeBuild, deployment is automatic:</p><ol><li>Developer pushes code to <strong>CodeCommit</strong>.</li><li><strong>CodePipeline</strong> is triggered by a CloudWatch Event Rule.</li><li><strong>CodeBuild</strong>:</li></ol><ul><li>Pulls source</li><li>Builds Docker image</li><li>Pushes to ECR</li><li>SSHs into EC2</li><li>Runs a deploy script to stop &amp; restart the container</li></ul><p>Here’s a snippet of what CodeBuild does:</p><pre>aws secretsmanager get-secret-value --secret-id MyKeyPem --query SecretString --output text &gt; ec2.pem<br>chmod 400 ec2.pem<br>ssh -o StrictHostKeyChecking=no -i ec2.pem ec2-user@$SERVER_IP \<br>  &quot;bash -s &lt; /home/ec2-user/deploy.sh $REPO_URI $IMAGE_TAG $SERVICE &#39;-p $PORT:$PORT&#39; &#39;-e NODE_ENV=$NODE_ENV&#39;&quot;</pre><h3>🧾 Full CloudFormation Template</h3><p>Below is the <strong>CloudFormation template</strong> that provisions the full CI/CD stack for building Docker images and deploying them to an EC2 instance:</p><pre>AWSTemplateFormatVersion: 2010-09-09<br><br>Description: &gt;<br>  Optimized AWS CloudFormation template for setting up a CI/CD pipeline using AWS CodePipeline and CodeBuild with deployment to Amazon EC2. This version improves security, modularity, and maintainability.<br><br>Parameters:<br>  ServiceName:<br>    Type: String<br>    Default: MyService<br>  ContainerRegistery:<br>    Type: String<br>    Default: my-docker-repo-name<br>  RepositoryName:<br>    Type: String<br>    Default: my-repo-name<br>  BranchName:<br>    Type: String<br>    Default: release<br>    AllowedPattern: ^[\w\-]+$<br>    ConstraintDescription: Must be alphanumeric with dashes or underscores.<br>  PortName:<br>    Type: String<br>    Default: &quot;1339&quot;<br>  Environment:<br>    Type: String<br>    Default: PROD<br>    AllowedPattern: &#39;[a-zA-Z0-9\-_]+&#39;<br>    ConstraintDescription: Must be alphanumeric with dashes or underscores.<br>  ArtifactBucketStore:<br>    Type: String<br>    Default: &quot;&quot;<br>  EnvSecretsArn:<br>    Type: String<br>    Description: ARN of SecretsManager secret containing environment variables<br>    Default: &quot;MYAPPS/PROD&quot;<br>  EC2PublicIp:<br>    Type: String<br>    Description: Public IP of the target EC2 instance for deployment<br>    Default: &quot;x.x.x.x&quot;<br><br>Conditions:<br>  UseStackNameForBucket:<br>    Fn::Equals:<br>      - Ref: ArtifactBucketStore<br>      - &quot;&quot;<br><br>Resources:<br><br>  PipelineBucket:<br>    Type: AWS::S3::Bucket<br>    Condition: UseStackNameForBucket<br>    Properties:<br>      BucketName: !Ref AWS::StackName<br>      Tags:<br>        - Key: Environment<br>          Value: !Ref Environment<br>        - Key: Name<br>          Value: !Join [&#39;-&#39;, [!Ref Environment, !Ref AWS::StackName]]<br><br>  AmazonCloudWatchEventRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      AssumeRolePolicyDocument:<br>        Version: 2012-10-17<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service: events.amazonaws.com<br>            Action: sts:AssumeRole<br>      Path: /<br>      Policies:<br>        - PolicyName: CloudWatchStartPipeline<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Effect: Allow<br>                Action: codepipeline:StartPipelineExecution<br>                Resource: !Sub arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${AWS::StackName}<br><br>  CodeBuildServiceRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      Path: /<br>      AssumeRolePolicyDocument:<br>        Version: 2012-10-17<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service: codebuild.amazonaws.com<br>            Action: sts:AssumeRole<br>      ManagedPolicyArns:<br>        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser<br>        - arn:aws:iam::aws:policy/CloudWatchLogsFullAccess<br>        - arn:aws:iam::aws:policy/AmazonS3FullAccess<br>      Policies:<br>        - PolicyName: CustomSecretsAccess<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Effect: Allow<br>                Action:<br>                  - secretsmanager:GetSecretValue<br>                Resource: !Sub arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:*DevOpsKeyPair*<br><br>  CodePipelineServiceRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      Path: /<br>      AssumeRolePolicyDocument:<br>        Version: 2012-10-17<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service: codepipeline.amazonaws.com<br>            Action: sts:AssumeRole<br>      ManagedPolicyArns:<br>        - arn:aws:iam::aws:policy/AWSCodePipelineFullAccess<br>        - arn:aws:iam::aws:policy/AWSCodeBuildAdminAccess<br>        - arn:aws:iam::aws:policy/AmazonEC2ContainerServiceFullAccess<br><br>  AmazonCloudWatchEventRule:<br>    Type: AWS::Events::Rule<br>    Properties:<br>      EventPattern:<br>        source:<br>          - aws.codecommit<br>        detail-type:<br>          - &quot;CodeCommit Repository State Change&quot;<br>        resources:<br>          - !Sub arn:aws:codecommit:${AWS::Region}:${AWS::AccountId}:${RepositoryName}<br>        detail:<br>          event:<br>            - referenceCreated<br>            - referenceUpdated<br>          referenceType:<br>            - branch<br>          referenceName:<br>            - !Ref BranchName<br>      Targets:<br>        - Arn: !Sub arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${AWS::StackName}<br>          RoleArn: !GetAtt AmazonCloudWatchEventRole.Arn<br>          Id: codepipeline-AppPipeline<br><br>  CodeBuildProject:<br>    Type: AWS::CodeBuild::Project<br>    Properties:<br>      Name: !Ref AWS::StackName<br>      Artifacts:<br>        Type: CODEPIPELINE<br>      Environment:<br>        PrivilegedMode: true<br>        ComputeType: BUILD_GENERAL1_SMALL<br>        Image: aws/codebuild/standard:7.0<br>        Type: LINUX_CONTAINER<br>        EnvironmentVariables:<br>          - Name: AWS_DEFAULT_REGION<br>            Value: !Ref AWS::Region<br>          - Name: SERVICE_SERVICE<br>            Value: !Ref ServiceName<br>          - Name: REPOSITORY_URI<br>            Value: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ContainerRegistery}<br>          - Name: NODE_ENV<br>            Value: !Ref Environment<br>          - Name: PORT_NAME<br>            Value: !Ref PortName<br>          - Name: ENV_FILE<br>            Type: SECRETS_MANAGER<br>            Value: !Ref EnvSecretsArn<br>          - Name: SERVER_IP<br>            Value: !Ref EC2PublicIp<br>      Source:<br>        Type: CODEPIPELINE<br>        BuildSpec: |<br>          version: 0.2<br>          phases:<br>            pre_build:<br>              commands:<br>                - echo Logging in to Amazon ECR...<br>                - aws ecr get-login-password --region ap-southeast-1 | docker login --username AWS --password-stdin 802368621649.dkr.ecr.ap-southeast-1.amazonaws.com<br>                - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)<br>                - IMAGE_TAG=$NODE_ENV-${COMMIT_HASH:=latest}<br>                - echo &quot;Installing SSH tools&quot;<br>                - apt-get update &amp;&amp; apt-get install -y openssh-client jq<br>                - echo &quot;Fetching PEM key from Secrets Manager&quot;<br>                - aws secretsmanager get-secret-value --secret-id DevOpsKeyPair --query SecretString --output text &gt; ec2.pem<br>                - chmod 400 ec2.pem<br>                - echo &quot;Logging in to Amazon ECR&quot;                <br>            build:<br>              commands:<br>                - aws secretsmanager get-secret-value --secret-id $SECRETS_MANAGER --region ap-southeast-1 --query SecretString --output text&gt; ./.env<br>                - cat .env<br>                - echo Build started on `date`<br>                - echo Building the Docker image...<br>                - docker build -t $REPOSITORY_URI:$IMAGE_TAG .<br>            post_build:<br>              commands:<br>                - echo Build completed on `date`<br>                - echo Pushing the Docker images...<br>                - docker push $REPOSITORY_URI:$IMAGE_TAG<br>                - echo &quot;Connecting to EC2 and managing Docker container&quot;<br>                - ssh -o StrictHostKeyChecking=no -i ec2.pem ec2-user@$SERVER_IP &quot;bash -s &lt; /home/ec2-user/deploy.sh $REPOSITORY_URI $IMAGE_TAG $SERVICE_SERVICE &#39;-p $PORT_NAME:$PORT_NAME&#39; &#39;-e NODE_ENV=$NODE_ENV&#39;&quot;<br>      ServiceRole: !Ref CodeBuildServiceRole<br><br>  Pipeline:<br>    Type: AWS::CodePipeline::Pipeline<br>    DependsOn:<br>      - CodeBuildProject<br>    Properties:<br>      RoleArn: !GetAtt CodePipelineServiceRole.Arn<br>      Name: !Ref AWS::StackName<br>      ArtifactStore:<br>        Type: S3<br>        Location: !If [UseStackNameForBucket, !Ref PipelineBucket, !Ref ArtifactBucketStore]<br>      Stages:<br>        - Name: Source<br>          Actions:<br>            - Name: SourceAction<br>              ActionTypeId:<br>                Category: Source<br>                Owner: AWS<br>                Version: 1<br>                Provider: CodeCommit<br>              Configuration:<br>                RepositoryName: !Ref RepositoryName<br>                BranchName: !Ref BranchName<br>                PollForSourceChanges: false<br>              OutputArtifacts:<br>                - Name: App<br>        - Name: DockerImageBuild<br>          Actions:<br>            - Name: Build<br>              ActionTypeId:<br>                Category: Build<br>                Owner: AWS<br>                Version: 1<br>                Provider: CodeBuild<br>              Configuration:<br>                ProjectName: !Ref CodeBuildProject<br>              InputArtifacts:<br>                - Name: App<br>              OutputArtifacts:<br>                - Name: BuildOutput<br><br>Outputs:<br>  PipelineName:<br>    Description: The name of the created CodePipeline<br>    Value: !Ref Pipeline<br>  ArtifactBucket:<br>    Description: Bucket used to store artifacts<br>    Value: !If [UseStackNameForBucket, !Ref AWS::StackName, !Ref ArtifactBucketStore]<br>  CodeBuildProject:<br>    Description: CodeBuild project name<br>    Value: !Ref CodeBuildProject</pre><p>This template handles:</p><ul><li>IAM roles</li><li>CloudWatch trigger setup</li><li>CodeBuild project (including environment vars and build spec)</li><li>CodePipeline definition</li><li>Optional S3 artifact bucket</li><li>SecretsManager integration for secure credential handling</li></ul><h3>✅ Results</h3><ul><li>✅ <strong>~70% cost reduction</strong> in non-production environments</li><li>✅ Centralized deployment &amp; logs via CodePipeline</li><li>✅ Full automation, minimal infrastructure</li><li>✅ Easy to scale: Just change the port and route</li></ul><h3>📌 Final Thoughts</h3><p>This approach may not be suitable for production environments that need HA, auto-scaling, or complex networking. But for <strong>development and QA environments</strong>, it’s a game-changer in terms of <strong>cost, simplicity, and control</strong>.</p><p>If you’re running ECS for non-production services and watching costs spiral, I highly recommend giving this architecture a try. It’s infrastructure-light, easy to manage, and still 100% AWS-native.</p><p><strong>Have questions or want the GitHub version of the template? Drop a comment below!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa3ffff40572" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/from-ecs-to-ec2-how-i-slashed-aws-costs-for-non-production-environments-with-codepipeline-and-fa3ffff40572">From ECS to EC2: How I Slashed AWS Costs for Non-Production Environments with CodePipeline and…</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Distributed Tracing with Jaeger DaemonSet: A Comprehensive Guide to Log Collection]]></title>
            <link>https://medium.datadriveninvestor.com/optimizing-distributed-tracing-with-jaeger-daemonset-a-comprehensive-guide-to-log-collection-1963cebee37?source=rss-bb9ba485dc06------2</link>
            <guid isPermaLink="false">https://medium.com/p/1963cebee37</guid>
            <category><![CDATA[jaeger]]></category>
            <category><![CDATA[jaegertracing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Pyae Phyoe Shein]]></dc:creator>
            <pubDate>Mon, 28 Apr 2025 14:58:53 GMT</pubDate>
            <atom:updated>2025-04-30T07:25:36.031Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hXLX76DNfEDtoTt8rbRmoA.jpeg" /><figcaption>image from: <a href="https://logz.io/blog/jaeger-tracing-performance/">https://logz.io/blog/jaeger-tracing-performance</a></figcaption></figure><h3>Introduction</h3><p>In today’s microservices architecture, understanding how requests flow through your system is crucial for debugging, performance optimization, and system reliability. Distributed tracing has emerged as an essential observability tool, with Jaeger being one of the most popular open-source solutions. In this article, we’ll explore how to effectively deploy Jaeger as a DaemonSet in Kubernetes to collect logs and traces efficiently across your entire cluster.</p><h4><em>Disclaimer</em></h4><p><em>I developed this Jaeger Helm template three years ago while working in Thailand. It was created for personal use to facilitate log tracing for services within Kubernetes environments. Please note that I cannot guarantee whether this template has been updated since its creation.</em></p><h3>Understanding Jaeger Architecture</h3><p>Before diving into the DaemonSet implementation, let’s briefly review Jaeger’s architecture:</p><ol><li>Jaeger Client: Libraries integrated into your applications that create spans</li><li>Jaeger Agent: A network daemon that collects spans and forwards them to the collector</li><li>Jaeger Collector: Validates, processes, and stores traces in the backend storage</li><li>Query Service: Interface for retrieving and visualizing traces</li><li>Storage Backend: Database for storing trace data (Elasticsearch, Cassandra, etc.)</li></ol><h3>Why Use a DaemonSet for Jaeger Agents?</h3><p>A DaemonSet ensures that a copy of a pod runs on each node in your Kubernetes cluster. For Jaeger agents, this approach offers several advantages:</p><ol><li>Reduced Network Overhead: Applications send traces to the local agent on the same node, minimizing network hops</li><li>Improved Reliability: If one agent fails, only traces from that node are affected</li><li>Better Performance: Local buffering and batching of spans before sending to the collector</li><li>Simplified Configuration: Applications only need to know about the local agent, not the collector</li></ol><h3>Implementing Jaeger Agent as a DaemonSet</h3><p>Let’s walk through setting up a Jaeger Agent DaemonSet in Kubernetes using Helm:</p><h4>Step 1: Create a Helm Chart Structure</h4><pre>jaeger-agent/<br>├── Chart.yaml<br>├── values.yaml<br>└── templates/<br>    ├── daemonset.yaml<br>    ├── service.yaml<br>    └── _helpers.tpl</pre><h4>Step 2: Define the DaemonSet</h4><p>The core of our implementation is the DaemonSet definition:</p><pre>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: jaeger-agent<br>spec:<br>  selector:<br>    matchLabels:<br>      app: jaeger-agent<br>  template:<br>    metadata:<br>      labels:<br>        app: jaeger-agent<br>    spec:<br>      containers:<br>      - name: jaeger-agent<br>        image: jaegertracing/jaeger-agent:1.35.0<br>        args:<br>          - &quot;--reporter.grpc.host-port=jaeger-collector:14250&quot;<br>          - &quot;--log-level=debug&quot;<br>        ports:<br>          - containerPort: 5775<br>            protocol: UDP<br>            name: zipkin-compact<br>          - containerPort: 6831<br>            protocol: UDP<br>            name: jaeger-compact<br>          - containerPort: 6832<br>            protocol: UDP<br>            name: jaeger-binary<br>          - containerPort: 5778<br>            protocol: TCP<br>            name: config<br>        resources:<br>          limits:<br>            memory: 200Mi<br>          requests:<br>            memory: 100Mi</pre><p>This configuration deploys the Jaeger agent on every node, listening on standard ports for different trace formats.</p><h3>Configuring Log Collection</h3><p>Jaeger primarily collects distributed traces, but we can enhance it to collect logs as well. Here are two approaches:</p><h4>1. Using the Jaeger Agent for Log Correlation</h4><p>While Jaeger itself doesn’t collect logs directly, we can correlate logs with traces by including trace IDs in log entries:</p><pre>env:<br>  - name: JAEGER_SERVICE_NAME<br>    value: &quot;my-service&quot;<br>  - name: JAEGER_PROPAGATION<br>    value: &quot;b3&quot;<br>  - name: JAEGER_SAMPLER_TYPE<br>    value: &quot;const&quot;<br>  - name: JAEGER_SAMPLER_PARAM<br>    value: &quot;1&quot;</pre><h4>2. Integrating with a Logging Solution</h4><p>For comprehensive log collection alongside traces, integrate Jaeger with a logging stack:</p><pre># Add a sidecar container for log forwarding<br>- name: log-forwarder<br>  image: fluent/fluent-bit:1.9<br>  volumeMounts:<br>    - name: log-config<br>      mountPath: /fluent-bit/etc/<br>    - name: log-storage<br>      mountPath: /var/log/containers/<br>  # Configuration to add trace IDs to logs</pre><h3>Best Practices for Jaeger DaemonSet Deployment</h3><ol><li>Resource Limits: Set appropriate CPU and memory limits to prevent agents from consuming excessive resources</li><li>Sampling Strategy: Configure intelligent sampling to reduce the volume of traces in high-traffic environments</li><li>Buffer Sizing: Adjust queue sizes based on your traffic patterns to prevent data loss during spikes</li><li>Security: Use network policies to restrict who can send spans to the agent</li><li>Monitoring: Set up alerts for agent health and backpressure metrics</li></ol><h3>Performance Tuning</h3><p>To optimize your Jaeger DaemonSet performance:</p><pre>args:<br>  - &quot;--processor.jaeger-binary.workers=10&quot;<br>  - &quot;--processor.jaeger-compact.workers=10&quot;<br>  - &quot;--processor.zipkin-compact.workers=10&quot;<br>  - &quot;--reporter.queue-size=10000&quot;<br>  - &quot;--reporter.batch-size=1000&quot;<br>  - &quot;--reporter.batch-delay=5s&quot;</pre><p>These settings increase the number of worker threads processing spans and optimize batching for sending to the collector.</p><h3>Handling High-Volume Environments</h3><p>For high-volume environments, consider these additional configurations:</p><ol><li>Adaptive Sampling: Implement dynamic sampling rates based on traffic volume</li><li>Collector Scaling: Ensure your collector deployment scales horizontally</li><li>Storage Optimization: Choose appropriate storage backends (Elasticsearch for searchability, Cassandra for scale)</li><li>Span Filtering: Filter out less important spans at the agent level</li></ol><h3>Monitoring Your Jaeger DaemonSet</h3><p>Monitor the health of your Jaeger agents with Prometheus metrics:</p><pre># Add to your DaemonSet spec<br>args:<br>  - &quot;--metrics-backend=prometheus&quot;<br>  - &quot;--metrics-http-route=/metrics&quot;<br>ports:<br>  - containerPort: 14271<br>    name: admin-http</pre><p>Key metrics to watch:</p><ul><li>jaeger_agent_reporter_batches_failures</li><li>jaeger_agent_reporter_queue_length</li><li>jaeger_agent_reporter_spans_failures</li></ul><h3>Conclusion</h3><p>Deploying Jaeger as a DaemonSet in Kubernetes provides an efficient way to collect distributed traces across your entire cluster. By following the best practices outlined in this article, you can ensure reliable trace collection while minimizing resource usage and network overhead.</p><p>Remember that while Jaeger excels at trace collection, comprehensive observability requires integrating traces with logs and metrics. By correlating these three pillars of observability, you’ll gain deeper insights into your system’s behavior and performance.</p><p>Have you implemented Jaeger in your Kubernetes environment? What challenges did you face? Share your experiences in the comments below!</p><p><em>This article was written by a distributed systems enthusiast with experience implementing observability solutions in cloud-native environments. For more content on Kubernetes, observability, and microservices architecture, follow me on Medium.</em></p><p>Visit us at <a href="https://www.datadriveninvestor.com/"><em>DataDrivenInvestor.com</em></a></p><p>Subscribe to DDIntel <a href="https://www.ddintel.com/"><em>here</em></a>.</p><p>Join our creator ecosystem <a href="https://join.datadriveninvestor.com/"><em>here</em></a>.</p><p>DDI Official Telegram Channel: <a href="https://t.me/+tafUp6ecEys4YjQ1">https://t.me/+tafUp6ecEys4YjQ1</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1963cebee37" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/optimizing-distributed-tracing-with-jaeger-daemonset-a-comprehensive-guide-to-log-collection-1963cebee37">Optimizing Distributed Tracing with Jaeger DaemonSet: A Comprehensive Guide to Log Collection</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>