<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Duy Hưng on Medium]]></title>
        <description><![CDATA[Stories by Duy Hưng on Medium]]></description>
        <link>https://medium.com/@tdhtp2016?source=rss-ed98d4be1205------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wByV73go_0nBAEIWvQUVvg.png</url>
            <title>Stories by Duy Hưng on Medium</title>
            <link>https://medium.com/@tdhtp2016?source=rss-ed98d4be1205------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:31:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tdhtp2016/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Self host Maybe Finance app on AWS with Amazon EC2]]></title>
            <link>https://medium.com/@tdhtp2016/self-host-maybe-finance-app-on-aws-with-amazon-ec2-e08243fdba6c?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/e08243fdba6c</guid>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[maybe]]></category>
            <category><![CDATA[ec2]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Mon, 21 Jul 2025 09:01:28 GMT</pubDate>
            <atom:updated>2025-07-22T13:53:19.082Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jAa3yT5ziTWyF9Vh.png" /></figure><p>Maybe is a open-source personal finance management application. In this blog, I will guide you to deploy an Maybe app for your own usage.</p><p>Github repo: <a href="https://github.com/maybe-finance/maybe">maybe-finance/maybe: The personal finance app for everyone</a></p><p>Website: <a href="https://maybefinance.com/">Maybe Finance: The personal finance app for everyone</a></p><p>Maybe Finance is a Ruby on Rails application that requires:</p><ul><li>PostgreSQL database</li><li>Redis for background jobs</li><li>Rails web server</li><li>Sidekiq worker process</li></ul><h4>Prerequisites</h4><p>Before we start, I assume that you already had these knowledges:</p><ul><li>AWS Account with appropriate permissions</li><li>Basic knowledge of AWS EC2 and Docker</li></ul><p>Here are the steps:</p><p>1 . Launch EC2 Instance</p><p>Launch Ubuntu 24.04 LTS instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*h2X76RfxwMwTIxlcef2O2w.png" /></figure><p>Recommended: t3.medium or larger (2 vCPU, 4GB RAM minimum).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*5tIGgwUiSIuHNyGPun5zPA.png" /></figure><p>Storage: 15 GiB gp3 storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*-Lj8MuZ8gEAFdenNcWNQcA.png" /></figure><p>Inbound Rules:</p><ul><li>SSH (22): Your IP or anywhere 0.0.0.0/0</li><li>Custom TCP (3000): 0.0.0.0/0 (for direct access to web app)</li><li>HTTP (80): 0.0.0.0/0 (if using ALB)</li><li>HTTPS (443): 0.0.0.0/0 (if using ALB)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*d1K4z7MaRAmhYT-50SAZ7A.png" /></figure><p>Remember to also put your instance in a public subnet.</p><p>2 . Once your intance ready, connect to your EC2 instance.</p><p>You can use EC2 instance connect or directly ssh.</p><pre>ssh -i your-key.pem ubuntu@your-ec2-ip</pre><p>Update instance’s system.</p><pre>sudo apt update &amp;&amp; sudo apt upgrade -y</pre><p>Install Docker, we need to run an image for the app.</p><pre>curl -fsSL https://get.docker.com -o get-docker.sh<br>sudo sh get-docker.sh</pre><p>When Docker is installed, it creates a special group called docker. By default, only the root user can run Docker commands. To allow regular users to run Docker commands without using sudo every time, you need to add them to the docker group.</p><pre>sudo usermod -aG docker ubuntu</pre><p>Install Docker Compose. The chmod +x step is essential for making any downloaded binary executable on Linux systems.</p><pre>sudo curl -L &quot;https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)&quot; -o /usr/local/bin/docker-compose<br>sudo chmod +x /usr/local/bin/docker-compose</pre><p>Logout and login again for docker group to take effect.</p><pre>exit</pre><p>3 . Set up Maybe Finance app.</p><p>Create application directory</p><pre>mkdir -p ~/maybe-finance<br>cd ~/maybe-finance</pre><p>Download the Docker Compose file.</p><pre>curl -o compose.yml https://raw.githubusercontent.com/maybe-finance/maybe/main/compose.example.yml</pre><p>Have a litte look in the compose file. You can change user, password and db if you want to.</p><pre>x-db-env: &amp;db_env<br>  POSTGRES_USER: ${POSTGRES_USER:-maybe_user}<br>  POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-maybe_password}<br>  POSTGRES_DB: ${POSTGRES_DB:-maybe_production}<br><br>x-rails-env: &amp;rails_env<br>  &lt;&lt;: *db_env<br>  SECRET_KEY_BASE: ${SECRET_KEY_BASE:-a7523c3d0ae56415046ad8abae168d71074a79534a7062258f8d1d51ac2f76d3c3bc86d86b6b0b307df30d9a6a90a2066a3fa9e67c5e6f374dbd7dd4e0778e13}<br>  SELF_HOSTED: &quot;true&quot;<br>  RAILS_FORCE_SSL: &quot;false&quot;<br>  RAILS_ASSUME_SSL: &quot;false&quot;<br>  DB_HOST: db<br>  DB_PORT: 5432<br>  REDIS_URL: redis://redis:6379/1<br># NOTE: enabling OpenAI will incur costs when you use AI-related features in the app (chat, rules).  Make sure you have set appropriate spend limits on your account before adding this.<br>  OPENAI_ACCESS_TOKEN: ${OPENAI_ACCESS_TOKEN}<br><br>services:<br>  web:<br>    image: ghcr.io/maybe-finance/maybe:latest<br>    volumes:<br>      - app-storage:/rails/storage<br>    ports:<br>      - 3000:3000<br>    restart: unless-stopped<br>    environment:<br>      &lt;&lt;: *rails_env<br>    depends_on:<br>      db:<br>        condition: service_healthy<br>      redis:<br>        condition: service_healthy<br>    networks:<br>      - maybe_net<br><br>  worker:<br>    image: ghcr.io/maybe-finance/maybe:latest<br>    command: bundle exec sidekiq<br>    restart: unless-stopped<br>    depends_on:<br>      redis:<br>        condition: service_healthy<br>    environment:<br>      &lt;&lt;: *rails_env<br>    networks:<br>      - maybe_net<br><br>  db:<br>    image: postgres:16<br>    restart: unless-stopped<br>    volumes:<br>      - postgres-data:/var/lib/postgresql/data<br>    environment:<br>      &lt;&lt;: *db_env<br>    healthcheck:<br>      test: [ &quot;CMD-SHELL&quot;, &quot;pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB&quot; ]<br>      interval: 5s<br>      timeout: 5s<br>      retries: 5<br>    networks:<br>      - maybe_net<br><br>  redis:<br>    image: redis:latest<br>    restart: unless-stopped<br>    volumes:<br>      - redis-data:/data<br>    healthcheck:<br>      test: [ &quot;CMD&quot;, &quot;redis-cli&quot;, &quot;ping&quot; ]<br>      interval: 5s<br>      timeout: 5s<br>      retries: 5<br>    networks:<br>      - maybe_net<br><br>volumes:<br>  app-storage:<br>  postgres-data:<br>  redis-data:<br><br>networks:<br>  maybe_net:<br>    driver: bridge</pre><p>Create environment file.</p><pre>cat &gt; .env &lt;&lt; EOF<br>POSTGRES_USER=maybe_user<br>POSTGRES_PASSWORD=$(openssl rand -base64 32)<br>POSTGRES_DB=maybe_production<br>SECRET_KEY_BASE=$(openssl rand -hex 64)<br>EOF</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnbdUMkwb6EemJ_iGEWBRw.png" /></figure><p>You can also set OPENAI_ACCESS_TOKEN to use Maybe AI in your app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*gyUdZYdzhenc50d3OJYSGw.png" /></figure><p>4 . Start the application</p><pre>docker-compose up -d</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*PP1UCVGkzuWzW8uH-xhtgQ.png" /></figure><p>Check if services are running</p><pre>docker-compose ps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ExV-MMN-CLLusxZBIe5kQ.png" /></figure><p>View logs</p><pre>docker-compose logs -f web</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*iVL8ez_a3PTztLj-zZJxpQ.png" /></figure><p>5 . Access your app and start using</p><p>Direct access on port 3000: <a href="http://your-ec2-public-ip:3000/">http://your-ec2-public-ip:3000</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nfm9qe879RqNH9PB-baaw.png" /></figure><p>Create your first account by clicking “Create your account” on the login page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPJth9Wn-h53vaCrnBiCgg.png" /></figure><blockquote>You can stop the instance when not necessary. When you need to use the app, just start the instance again and use.</blockquote><p>Explore the app yourself and manage your finance effectively!</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏️<strong>️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e08243fdba6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a simple collaborative filtering recommendation engine with Apache Spark ML on Amazon EMR…]]></title>
            <link>https://aws.plainenglish.io/build-a-simple-collaborative-filtering-recommendation-engine-with-apache-spark-ml-on-amazon-emr-500d3caf3331?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/500d3caf3331</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[amazon-emr]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[als]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Mon, 16 Jun 2025 16:24:07 GMT</pubDate>
            <atom:updated>2025-06-30T07:08:39.251Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build a simple collaborative filtering recommendation engine with Apache Spark ML on Amazon EMR Studio</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3C1JpBrfYbx4PoXSXxvvLg.png" /></figure><h3>Introduction</h3><p><strong>Amazon EMR</strong> (Elastic MapReduce) is a fully managed, cloud-native service from AWS that makes it easy to process and analyze large amounts of data using open-source big data frameworks such as <strong>Apache Hadoop, Apache Spark, Apache Hive, Apache HBase, Apache Flink, Presto, Trino,.</strong>. It allows you to quickly and cost-effectively run big data frameworks in a scalable and flexible manner on the AWS Cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*Y4jo0FyXNU35bvaU2FNAjw.png" /></figure><p>Amazon EMR is the best place to run Apache Spark. Amazon EMR Studio is a web-based integrated development environment (IDE) for fully managed Jupyter notebooks that run on Amazon EMR clusters. You can set up an EMR Studio for your team to develop, visualize, and debug applications written in R, Python, Scala, and PySpark.</p><p>Read more about Amazon EMR Studio at: <a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio.html">Amazon EMR Studio — Amazon EMR</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*o6nsDnXPDc_dV8OASz8JfQ.png" /></figure><p><a href="https://spark.apache.org/docs/latest/api/python/index.html">PySpark </a>is the Python API for Apache Spark. It enables you to perform real-time, large-scale data processing in a distributed environment using Python..</p><p><a href="https://spark.apache.org/docs/latest/ml-guide.html">Apache Spark MLlib</a> is Spark’s machine learning (ML) library. Its goal is to make practical machine learning scalable and easy.</p><p><strong>Collaborative Filtering </strong>is a <strong>technique</strong> used in recommendation systems to make predictions about a user’s interests by collecting preferences from many users. It is one of the most popular methods used in <strong>recommender systems</strong>. The core idea is that if users have similar preferences in the past, they will likely have similar preferences in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WMWlDcutvuYStfzJ" /></figure><p>We will use <strong>Collaborative Filtering from </strong><a href="https://spark.apache.org/docs/latest/ml-guide.html">Apache Spark MLlib</a> using <strong>ALS (Alternating Least Squares)</strong>, read more about that here:</p><p><a href="https://spark.apache.org/docs/latest/ml-collaborative-filtering.html">Collaborative Filtering — Spark 4.0.0 Documentation</a></p><h3>Process</h3><p>Before we start, I assume that you have an AWS account and some basic knowledges about AWS enviroment.</p><p><strong>Also here is the cost after I let the resources run for more than 1 day:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fkgs_OelVEMYkQeRhIr66w.png" /></figure><p><strong>You should consider the cost before you start!</strong></p><p>1 . Navigate to Amazon EMR, choose <strong>Amazon EMR Studio </strong>then <strong>Create studio</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4v1LRJolq_i_5SGIObfEPQ.png" /></figure><p>2 . Choose <strong>Interactive workloads , then choose Edit </strong>in <strong>Studio settings.</strong></p><p>Enter <strong>EMR_Studio </strong>for <strong>Studio name.</strong></p><p>Choose to create new bucket to store waorkspace package.</p><p>Also create a new service role<strong> AmazonEMRStudio_ServiceRole.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ndQSX43rhniYyqEu-20WMQ.png" /></figure><p>3 . Enter <strong>Collaborative_Filtering_Recommendation </strong>for workspace name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*R_7QeTW6Pf6J0HCpLfCNBA.png" /></figure><p>4 . Enter <strong>Serverless_Interactive_App </strong>for application name.</p><p>Create a new runtime role <strong>AmazonEMRStudio_RuntimeRole</strong>, this role is important.</p><p>Then create studio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-jsbjlK08L7n0uinYzWag.png" /></figure><p>You can expand the default settings for interactive workloads to see the details of configs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1QPCyZATTlHG7nownGI5jQ.png" /></figure><p>5 . THIS PART IS IMPORTANT!</p><p>Go to IAM and search for the runtime role <strong>AmazonEMRStudio_RuntimeRole.</strong></p><p>We need to attach the <strong>S3FullAccess</strong> to this role so that it can read and write object in S3 buckets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LiL8560TVAKuEEXQkHs6Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*Rip2GrAix3sKdDCEWC631g.png" /></figure><p>Copy and save the ARN of the runtime role <strong>AmazonEMRStudio_RuntimeRole, </strong>we will need it later<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aylP6uvkDh_KRiJ4bg_5Dw.png" /></figure><p>6 . Back to EMR Studio, let’s try to run the notebook.1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6o4LhwYdzr36cu-Iq7ohQ.png" /></figure><p>You can see here that it had been successfully deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c87qB1N5QBhLmk_0ijArZA.png" /></figure><p>7 . Now you need to go to S3 and make a new bucket. Then go to permission and attach this policy to your bucket.</p><p>Remember to add the ARN of the runtime role. This policy give permission to the role so that it can access the bucket.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;Policy for Runtime Role&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::xxxxxxxxxxxx:role/service-role/AmazonEMRStudio_RuntimeRole&quot; # Replace with your runtime role<br>            },<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::emr-studio-blog/*&quot; # Replace with your bucket<br>        }<br>    ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PNXG9vRcpjU_vjRY7S0UXQ.png" /></figure><p>Then go make 3 folders <strong>data, model</strong>, and <strong>result</strong>.</p><p>Folder data will store the dataset in csv format.</p><p>Model will store the model after we finish the training.</p><p>Result will store the recommendation results from the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*JkxKI17nHqgYmX692jbDJA.png" /></figure><p>8 . Here is the code for this blog <a href="https://github.com/Hung-00/collaborative-filtering-recommendation-engine-Amazon-EMR">Hung-00/collaborative-filtering-recommendation-engine-Amazon-EMR: collaborative filtering recommendation engine with Apache Spark ML on Amazon EMR</a>.</p><p>Download the repo, have a look at it and upload the <strong>rating.csv</strong> to the data folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*GUuN7w5tezqieR3Vu9GvsA.png" /></figure><p>Then head back to the notebook and upload the <strong>ASL.ipynb</strong> that I made for this blog.</p><p>This notebook has every steps that you need to run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hf84KgpDrd8Rmb0nMfWGyQ.png" /></figure><p>9 . Open the notebook, make sure that you run a <strong>PySpark </strong>kernel for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*q10OgQp-zx-Q_WSRGQpq0g.png" /></figure><p>10 . Now you can go through every steps that I prepared.</p><p>In some steps, you need to change the S3 URI to reference to your bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HQrctPHHZEzjPyeHI5fTzQ.png" /></figure><p>Train a simple model with ALS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUAqGJ6QYmvnCN3y9mBtLg.png" /></figure><p>11 . Save the recommendation result to S3.</p><p>You will see two files; the first is an empty file named \_SUCCESS, denoting that the PySpark script completed and wrote data to the Amazon S3 bucket successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4-epOUhEn-Dd_UVEq6zpig.png" /></figure><p>Model is saved to S3, we can invoke it later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*iNjgb-KqcNvxtFnOr6mfTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GQcjCJ1miuM2A7LV96HQNQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LKBwUWBUxVhOZVayS4FaBQ.png" /></figure><p>12 . There are some OPTIONAL steps that I let you to deep dive if u want to, like tuning, deploying and monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*Y2H21PYxWeFhSwBu57HMhw.png" /></figure><p>13 . Remember to clean up after you finish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*KLVTlHD0dtWF3z_rSJCrOg.png" /></figure><p>This blog just give you an idea on how to build a model with Amazon EMR. You should go ahead and try with larger dataset, more complex tasks.</p><p>Here is a movies dataset that you can try to make a movie recommedating model: <a href="https://grouplens.org/datasets/movielens/latest/">https://grouplens.org/datasets/movielens/latest/</a></p><p>Thank you for reading!</p><p>Reference:</p><ol><li><a href="https://github.com/Hung-00/collaborative-filtering-recommendation-engine-Amazon-EMR">Hung-00/collaborative-filtering-recommendation-engine-Amazon-EMR: collaborative filtering recommendation engine with Apache Spark ML on Amazon EMR</a></li><li><a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio.html">Amazon EMR Studio — Amazon EMR</a></li><li><a href="https://spark.apache.org/docs/latest/ml-guide.html">Apache Spark MLlib</a></li><li><a href="https://spark.apache.org/docs/latest/api/python/index.html">PySpark Overview — PySpark 4.0.0 documentation</a></li><li><a href="https://spark.apache.org/docs/latest/ml-collaborative-filtering.html">Collaborative Filtering — Spark 4.0.0 Documentation</a></li><li>Movie datasets: <a href="https://grouplens.org/datasets/movielens/latest/">https://grouplens.org/datasets/movielens/latest/</a></li></ol><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=500d3caf3331" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/build-a-simple-collaborative-filtering-recommendation-engine-with-apache-spark-ml-on-amazon-emr-500d3caf3331">Build a simple collaborative filtering recommendation engine with Apache Spark ML on Amazon EMR…</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Simple Puzzle Games with Amazon Q CLI and PyGame]]></title>
            <link>https://aws.plainenglish.io/build-simple-puzzle-games-with-amazon-q-cli-and-pygame-3cd6c9344033?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cd6c9344033</guid>
            <category><![CDATA[amazon-q]]></category>
            <category><![CDATA[pygame]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[amazon-q-cli]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Fri, 06 Jun 2025 08:29:58 GMT</pubDate>
            <atom:updated>2025-07-08T01:58:15.554Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eyk8gY3ws-Hy9SQBoT8Ktg.png" /></figure><p><a href="https://community.aws/content/2xIoduO0xhkhUApQpVUIqBFGmAc/build-games-with-amazon-q-cli-and-score-a-t-shirt?trk=b085178b-f0cb-447b-b32d-bd0641720467&amp;sc_channel=el">Build Games with Amazon Q CLI </a>is a new challenge for AWS community. If you have any idea for your game, go and try it out right now.</p><p>Today in this blog, I will show you my ideas for creating some simple game with Pygame Library.</p><h4>1. Installation</h4><p>Instal Amazon Q CLI on Linux or Mac: <a href="https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/command-line-installing.html">https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/command-line-installing.html</a></p><p>Install Amazon Q CLI on Window:<br><a href="https://community.aws/content/2v5PptEEYT2y0lRmZbFQtECA66M/the-essential-guide-to-installing-amazon-q-developer-cli-on-windows?trk=e07eca93-fa2f-4351-b567-f293b83eb635&amp;sc_channel=el_">https://community.aws/content/2v5PptEEYT2y0lRmZbFQtECA66M/the-essential-guide-to-installing-amazon-q-developer-cli-on-windows?trk=e07eca93-fa2f-4351-b567-f293b83eb635&amp;sc_channel=el_</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*tbINFFGwB735r2KCv-9yyQ.png" /></figure><p>Try some simple prompt to ensure that everything is good. If you see the question below, enter t to allow the action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bcQAb7yRnuSrCuRfoPWZHA.png" /></figure><p>If you are on Window, you can access WSL in File Explorer with this command:</p><pre>\\wsl$</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*Y97Zrc6kQhJejIcvSTlT-A.png" /></figure><p>You can copy file created from Amazon Q in this directory to your git project.</p><p>Install PyGame: <a href="https://www.pygame.org/wiki/GettingStarted">https://www.pygame.org/wiki/GettingStarted</a></p><p>Have a look at all the generated code and prompts that I used in this repo: <a href="https://github.com/Hung-00/Amazon-Q-CLI-Build-Games">Hung-00/Amazon-Q-CLI-Build-Games</a></p><p>Now let’s try creating some amazing games with Amazon Q CLI!</p><h4>2. Game 2048</h4><p><strong>First of all, 2048</strong> is a popular single-player puzzle game created by Italian web developer Gabriele Cirulli in 2014. The game is played on a 4x4 grid, where the goal is to combine numbered tiles by sliding them up, down, left, or right to create a tile with the number <strong>2048</strong>.</p><p>Now let’s recreate this game and add some idea with Amazon Q.</p><pre>2048 is a popular single-player puzzle game created by Gabriele Cirulli in 2014. <br>In this folder, I want you to create a 2048 game using PyGame library.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*cc4dbgigW-ZA-Z9gomRuqA.gif" /></figure><p>As you can see that the game created is kinda fine although it messed up some positioning for the instructions.</p><p>I used the prompt below to let Amazon Q know what it need to update the game.</p><pre>Here are something you need to update:<br>1. Update the grid to 8x8. Update the condition to win the game to 8192.<br>2. Increase the speed of tile&#39;s sliding effect. Change the code to wait for all sliding effects end then the new tile can appear.<br>3. Move all the title, how-to-play and score section to the left side of the window. Make sure this section not cover the game grid.<br>4. Add some simple sound effects.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pihfgpfnAa3VlUHJN6Tc9Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cFQy6qouseZgeSZ9me2vMg.png" /></figure><p>Amazon Q can not create wav file so you have to add it by yourself afterward. You can fine some simple game wav file easily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*FaEB-bcIVn9heXLg8jyA2Q.png" /></figure><p>And here is the final result:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUMaBOiPm7Pg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUMaBOiPm7Pg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FUMaBOiPm7Pg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d050480ede99a6f4deb6a24bfd4a339b/href">https://medium.com/media/d050480ede99a6f4deb6a24bfd4a339b/href</a></iframe><p>The game is now 8192 and in 8x8 grid, have sound effects and better layout.</p><p>It’s really amazing how Amazon Q can take requirements and improve the game pretty decently!</p><h4>3. Puzzle sliding game</h4><p>The <strong>15 Puzzle</strong> was invented by <strong>Noyes Palmer Chapman</strong> in 1874. It became a major craze in the United States during the 1880s, and its popularity spread quickly to Europe. It is one of the most well-known and challenging puzzles in history. The game consists of a 4x4 grid (16 slots) with 15 numbered tiles and one empty space, and the objective is to rearrange the tiles in a specific order, typically from 1 to 15, using the empty space to slide the tiles around.</p><p>Let’s start with this prompt.</p><pre>The slide puzzle, sometimes referred to as the &quot;15 puzzle&quot; or &quot;Gem Puzzle&quot;, was first created by Noyes Palmer Chapman in 1874. <br>Number Slide Puzzle (15 Puzzle) has a grid with numbered tiles where the player needs to order the tiles from 1 to the maximum number <br>(such as 1 to 15 in a 4x4 grid). Typically, the numbers are arranged in a sequence, but there&#39;s one empty space that allows the tiles to move. <br><br>In this folder, I want you to create a 2d Number Slide Puzzle (15 Puzzle) game using PyGame library.</pre><p>Below is the result, pretty simple but work kinda great.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQrMcXQ-eVBg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQrMcXQ-eVBg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQrMcXQ-eVBg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2960ff9f787c543fe83ba3080d4adbb5/href">https://medium.com/media/2960ff9f787c543fe83ba3080d4adbb5/href</a></iframe><p>I will give some idea to Amazon Q CLI through this prompt:</p><pre>Here are something you need to update the slide puzzle game:<br>   1. Give the player an option to choose 3x3, 4x4, 5x5 or 6x6 grid. Give me an option to go back to the main menu when I&#39;m in the game.<br>   2. In the main menu, give the player the option to choose the image for the grid, make the grid into image-based grid. You will split the picture into pieces and make them become the tiles in the playground. The bottom right pieces will be removed.<br>   3. Add some simple sound effects for sliding effects.</pre><p>Here is the response from Amazon Q CLI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ia3p4Wr9go7AmdWu4nghjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZyQKvmkbCTWEfO0lJTpyw.png" /></figure><p>You need to put some image in the same folder with the code so that the game can detect it. I’ve already put some image is the source code for you to try it right away.</p><p>And this is the result:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_qA9G0JsM4A&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_qA9G0JsM4A&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F_qA9G0JsM4A%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/805649ce3f1394d0ca7e8fbab8a2e0c4/href">https://medium.com/media/805649ce3f1394d0ca7e8fbab8a2e0c4/href</a></iframe><p>The game has become smoother, more beautiful, with more options and greater variety for the players.</p><p>It’s truly impressive how Amazon Q CLI can understand requirements, analyze and enhance the game effectively!</p><p>Amazon Q CLI even add some more feature in the that we don’t even know that we need it, which is really impressive!</p><h4>Other ideas</h4><p>I have tried some ideas and also include them in my code repo, go check them out!</p><p>Tetris</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*XnB1lHXJUcL3Wzl_1q9zeQ.png" /></figure><p>Ultimate Tic-Tac-Toe</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*Fa3vOmfu_VCGb85LnazTFQ.png" /></figure><h4>Conclusion</h4><p>You can get all the generated code and prompts that I used in this repo, feel free to try it out: <a href="https://github.com/Hung-00/Amazon-Q-CLI-Build-Games">Hung-00/Amazon-Q-CLI-Build-Games</a></p><p>Thank you and have a nice day!</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cd6c9344033" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/build-simple-puzzle-games-with-amazon-q-cli-and-pygame-3cd6c9344033">Build Simple Puzzle Games with Amazon Q CLI and PyGame</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Achieving planogram tasks with Amazon Bedrock and Amazon Nova effectively]]></title>
            <link>https://aws.plainenglish.io/achieving-planogram-tasks-with-amazon-bedrock-and-amazon-nova-effectively-85901a0421fd?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/85901a0421fd</guid>
            <category><![CDATA[nova]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-nova]]></category>
            <category><![CDATA[bedrock]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Mon, 05 May 2025 09:19:55 GMT</pubDate>
            <atom:updated>2025-07-06T18:15:23.992Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*nAEn9-hXEv5ARWMincAKuA.png" /></figure><p>Planogram management — the careful arrangement and optimization of products on retail shelves — is critical for maximizing sales, enhancing customer experience, and efficiently utilizing retail space. Traditionally, creating and updating planograms has been a manual, labor-intensive task, often prone to errors and inconsistencies. However, recent advancements in Large Language Models (LLMs) present an exciting opportunity to revolutionize how retailers approach this challenge.</p><p>In this blog post, I will show you how to use LLMs on Amazon Bedrock, especially Amazon Nova to explore how LLMs can automate and streamline planogram-related tasks. I will also try out some recommended prompting technique from AWS Nova Team.</p><h4>LLMs Preparations</h4><p>1 . Prepare LLM access and IAM Roles policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*5O_tzLnL82sn1NDSLUdnTA.png" /></figure><p>Make sure that you have access to some LLMs on Bedrock like Amazon Nova, Claude,… and set your role’s credentials on your environments with necessary policies like AdministratorAccess or BedrockFullAccess.</p><p>2 . Have a look at this repo: <a href="https://github.com/Hung-00/Amazon-Nova-Planogram-Example"><strong>Hung-00/Amazon-Nova-Planogram-Example</strong></a>. I have prepared the code to invoke some LLMs and some pictures to test.</p><p>3 . Read about Amazon Nova here: <a href="https://docs.aws.amazon.com/nova/latest/userguide/what-is-nova.html"><strong>What is Amazon Nova? — Amazon Nova</strong></a></p><p><a href="https://docs.aws.amazon.com/nova/latest/userguide/using-invoke-api.html">4 . Copy the code and try to invoke the LMMs : <strong>Using the Invoke API — Amazon Nova</strong></a><strong>. </strong>Make sure that you invole Nova successfully.</p><h4><strong>System Prompt Preparations</strong></h4><p>Amazon Bedrock provides us a wonderful feature — <strong>optimize prompt for different LLMs.</strong></p><p>Hop into playground, choose Nova Pro and try this prompt (you can also use the code in my repo):</p><pre>You a Planogram Specialist.<br>You will receive some pictures of shelves that have products on it.<br>You will need to identify the products, count the quantity and response to the user.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6dLsN7Opu_qyGPpsnHfpQ.png" /></figure><p>Bedrock will give you back the optimized prompt which is apropriate for the LLM you chose. In this case, Nova will have a beeter understanding and performance when receiving the optimized prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tg0DVieVw3W2nC9cpZGQ7w.png" /></figure><p>Here is another optimized prompt for Anthropic’s Claude 3 Sonnet. As you can see that Claude can do better with mark up prompts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZJIfnjITFawGSe4KTr-aQ.png" /></figure><p>Now you know how to get a good system prompts and use it in your applications.</p><p>You can also have a look at this repo <a href="https://github.com/jujumilk3/leaked-system-prompts"><strong>jujumilk3/leaked-system-prompts: Collection of leaked system prompts</strong></a><strong> </strong>for some system prompt syntax for LLMs.</p><h4>User Prompt Preparations</h4><p>Here are some tips to create a good user prompts for Nova:</p><p>1 . Pictures placements matter</p><p>Nova documents about this: <a href="https://docs.aws.amazon.com/nova/latest/userguide/prompting-vision-prompting.html#prompting-video-placement">https://docs.aws.amazon.com/nova/latest/userguide/prompting-vision-prompting.html#prompting-video-placement</a></p><p>When using Nova, you should always include the pictures in the array of user prompts.</p><p>For example, let’s try provides some pictures to Nova to analyze with zero-shot technique (you can find the code in <strong>nova_zeo_shot.py):</strong></p><pre>image = Path(&quot;ensure.jpg&quot;)<br><br>with open(image, &quot;rb&quot;) as image_file:<br>    evaluate_image_1 = base64.b64encode(image_file.read())<br><br>image = Path(&quot;chips.jpg&quot;)<br><br>with open(image, &quot;rb&quot;) as image_file:<br>    evaluate_image_2 = base64.b64encode(image_file.read())<br><br>message_list = [<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_1.decode()},<br>                }<br>            },<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_2.decode()},<br>                }<br>            },<br>            {<br>                &quot;text&quot;: &quot;&quot;&quot;<br>                Analyze the pictures. Seperate the answer for each picture.<br>                &quot;&quot;&quot;<br>            },<br>        ],<br>    }<br>]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TX_LpN3B_extAN77sdjMXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mkvlR50gW-OM5bC-_NTOg.png" /></figure><p>You can see that Nova did the job pretty good. With the optimized system prompts and we also provides the pictures before the prompts, Nova or any other LLMs can provide us the best answer.</p><p>Below is the answer I got from Claude 3.7 Sonnet, the result is very good like Nova:</p><pre>I&#39;ll analyze each of the product shelf images separately.<br><br>Image 1:<br><br>&lt;product_list&gt;<br>Ensure Gold StrengthPro: 3<br>Ensure Original Nutrition Shake (Vanilla flavor): 3<br>&lt;/product_list&gt;<br><br>Image 2:<br><br>&lt;product_list&gt;<br>Pringles Original: 3<br>Pringles Sour Cream &amp; Onion: 3<br>Pringles Cheesy Cheese: 2<br>Lay&#39;s Stax Sour Cream &amp; Onion: 2<br>Lay&#39;s Stax Hot Chili Squid: 2<br>Lay&#39;s Stax Original: 2<br>Lay&#39;s Stax Spicy Lobster: 2<br>&lt;/product_list&gt;</pre><p><em>There is only one down side that picture can cost a lots of token, 1 picture maybe equal to 1500 tokens. You need to be aware of this.</em></p><p><strong>Using few-shot technique for even better answer</strong></p><p>In the message list you can include as many content as you want to improve the model’s answer (have a look at<strong> nova_few_shot.py</strong> or <strong>claude_few_shot.py </strong>in my repo):</p><pre>image1 = Path(&quot;ensure.jpg&quot;)<br><br>with open(image1, &quot;rb&quot;) as image_file:<br>    evaluate_image_1 = base64.b64encode(image_file.read())<br><br>image2 = Path(&quot;xylitol.jpg&quot;)<br><br>with open(image2, &quot;rb&quot;) as image_file:<br>    evaluate_image_2 = base64.b64encode(image_file.read())<br><br>image3 = Path(&quot;chips.jpg&quot;)<br><br>with open(image3, &quot;rb&quot;) as image_file:<br>    evaluate_image_3 = base64.b64encode(image_file.read())<br><br><br>message_list = [<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_1.decode()},<br>                }<br>            },<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_2.decode()},<br>                }<br>            },<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_3.decode()},<br>                }<br>            },<br>            {<br>                &quot;text&quot;: &quot;&quot;&quot;<br>            The first picture has:<br>            - Product 1: Ensure Gold StrengthPro:, Quantity: 3<br>            - Product 2: Ensure Original Nutrition Shake (Vanilla flavor):, Quantity: 3<br>             &quot;&quot;&quot;<br>            },<br>            {<br>                &quot;text&quot;: &quot;&quot;&quot;<br>            The second picture has:<br>            - Product 1: Red Xylitol, Quantity: 2<br>            - Product 2: Purple Xylitol, Quantity: 6<br>            - Product 3: Blue Xylitol, Quantity: 4<br>             &quot;&quot;&quot;<br>            },<br>            {&quot;text&quot;: &quot;Analyze the third picture.&quot;},<br>        ],<br>    }<br>]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mbvWLsx5aP2B2TAPU4Z_fg.png" /></figure><p>You can also create a conversation with role so that LLMs can understand the situation (have a look at<strong> nova_few_shot_with_role.py</strong> or <strong>claude_few_shot_with_role.py </strong>in my repo).</p><pre>message_list = [<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_1.decode()},<br>                }<br>            },<br>        ],<br>    },<br>    {<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;text&quot;: &quot;&quot;&quot;<br>                - Product 1: Ensure Gold StrengthPro:, Quantity: 3<br>                - Product 2: Ensure Original Nutrition Shake (Vanilla flavor):, Quantity: 3<br>                &quot;&quot;&quot;,<br>            },<br>        ],<br>    },<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_2.decode()},<br>                }<br>            },<br>        ],<br>    },<br>    {<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;text&quot;: &quot;&quot;&quot;<br>                - Product 1: Red Xylitol, Quantity: 2<br>                - Product 2: Purple Xylitol, Quantity: 6<br>                - Product 3: Blue Xylitol, Quantity: 4<br>            &quot;&quot;&quot;,<br>            },<br>        ],<br>    },<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;image&quot;: {<br>                    &quot;format&quot;: &quot;jpeg&quot;,<br>                    &quot;source&quot;: {&quot;bytes&quot;: evaluate_image_3.decode()},<br>                }<br>            },<br>            {<br>                &quot;text&quot;: &quot;&quot;&quot;<br>                Analyze the third picture.<br>                &quot;&quot;&quot;,<br>            },<br>        ],<br>    },<br>]</pre><p>You can test it out in the code repo.</p><h3>Conclusion</h3><p>Amazon Nova is a new and really strong LLMs. Nova can handle complex tasks with good performance. Beside that, Nova also have many more features like Nova Canvs, Nova Reel, Nova Act,… The future of Amazon Nova and its eco system is very promising.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85901a0421fd" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/achieving-planogram-tasks-with-amazon-bedrock-and-amazon-nova-effectively-85901a0421fd">Achieving planogram tasks with Amazon Bedrock and Amazon Nova effectively</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bedrock Agents AIOps]]></title>
            <link>https://medium.com/@tdhtp2016/bedrock-agents-aiops-47a1d8dfe8c1?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/47a1d8dfe8c1</guid>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 13:28:37 GMT</pubDate>
            <atom:updated>2025-04-24T13:35:03.807Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bedrock Agents AIOps Part 1</h3><h3>How Bedrock Agents Work</h3><p>Agents for Amazon Bedrock consists of two main sets of API operations to help you set up and run an agent:</p><ul><li>Build-time API operations to create, configure, and manage your agents and their related resources</li><li>Runtime API operations to invoke your agent with user input and to initiate orchestration to carry out a task.</li></ul><h4>Build Time</h4><p>An agent consists of the following components:</p><ul><li><strong>Foundation model</strong> — You choose a foundation model (FM) that the agent invokes to interpret user input and subsequent prompts in its orchestration process. The agent also invokes the FM to generate responses and follow-up steps in its process.</li><li><strong>Instructions</strong> — You write instructions that describe what the agent is designed to do. With advanced prompts, you can further customize instructions for the agent at every step of orchestration and include Lambda functions to parse each step’s output.</li><li><strong>Action groups (Optional)</strong> — You define the actions that the agent should perform by providing the following resources:</li><li>An OpenAPI schema to define the API operations that the agent can invoke to perform its tasks.</li><li>A Lambda function with the following input and output:</li><li>Input — The API operation and parameters identified during orchestration.</li><li>Output — The result of the API invocation.</li><li><strong>Knowledge bases (Optional)</strong> — Associate knowledge bases with an agent. The agent queries the knowledge base for extra context to augment response generation and input into steps of the orchestration process.</li><li><strong>Prompt templates</strong> — Prompt templates form the foundation for prompts provided to the foundation model. Amazon Bedrock Agents include four default templates for pre-processing, orchestration, knowledge base responses, and post-processing. You can also edit or disable these templates.</li></ul><p>At build time, all these components combine to form the <strong>base prompts</strong> for agent orchestration. You can enhance these prompts with advanced logic or few-shot examples to improve step-by-step accuracy. Each prompt template includes <strong>instructions</strong> , <strong>action details</strong>, <strong>knowledge base references</strong>, and <strong>conversation history</strong>, all of which you can customize to suit your agent’s needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5xrRnVtxmZ1ZW6qN" /></figure><h4>Agent Runtime Process</h4><p>Runtime is managed by the InvokeAgent API operation. This operation starts the agent sequence, which consists of the following three main steps:</p><ul><li><strong>Pre-processing</strong> — Manages how the agent contextualizes and categorizes user input and can be used to validate input.</li><li><strong>Orchestration</strong> — Interprets the user input, invokes action groups and queries knowledge bases, and returns output to the user or as input to continued orchestration. Orchestration consists of the following steps:</li><li>The agent interprets the input with a foundation model and generates a rationale that lays out the logic for the next step it should take.</li><li>The agent invokes action groups and queries knowledge bases (Knowledge base response generation) to retrieve additional context and summarize the data to augment its generation.</li><li>The agent generates an output, known as an observation, from invoking action groups and summarizing results from knowledge bases. The agent uses the observation to augment the base prompt, which is then interpreted with a foundation model. The agent then determines if it needs to reiterate the orchestration process.</li><li><strong>Post-processing</strong> — The agent formats the final response to return to the user. This step is turned off by default.</li></ul><p>When you invoke your agent, you can turn on a trace at runtime. With the trace, you can track the agent’s rationale, actions, queries, and observations at each step of the agent sequence. The trace includes the full prompt sent to the foundation model at each step and the outputs from the foundation model, API responses, and knowledge base queries.</p><p>You can use the trace to understand the agent’s reasoning at each step. As the user session with the agent continues through more InvokeAgent requests, the conversation history is preserved. The conversation history continually augments the orchestration base prompt template with context, helping improve the agent’s accuracy and performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IslPeMpg9l77LrLo" /></figure><h3>Create your Agent</h3><h4><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/30-agent-setup-guide/32-create-your-agent#step-1:-navigate-to-bedrock-agents">Step 1: Navigate to Bedrock Agents</a></h4><p>1 . Navigate to Amazon Bedrock:</p><ul><li>Open the AWS Management Console</li><li>Search for and select <strong>Amazon Bedrock</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FNkRhHdtPdqTDaeH" /></figure><p>2 . Open the agent:</p><ul><li>Click the menu icon in the top left corner</li><li>Select <strong>Agents</strong></li><li>Choose <strong>Create agent</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*19MLt6zWa1zZOq2j" /></figure><p>3 . Provide the following input values:</p><ul><li><strong>Name</strong> : bedrock_agent</li><li><strong>Description</strong> : monitor system and give insight</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*NObrHb88Wkvx1ILtHPq4Og.png" /></figure><p>4 . Provide the following details.</p><p><strong>Agent resource role : Create and use a new service role</strong></p><p><strong>Select model: C</strong>hoose <strong>Anthropic</strong> and then select <strong>Claude 3.5 Sonnet V1 model</strong> in the dropdown menu.</p><p><strong>Instructions for the Agent: C</strong>opy and paste the following instruction below.</p><pre>&lt;core_guidelines&gt;<br>You are an AI-powered monitoring assistant ensuring clarity, accuracy, and real-time insights. <br>  &lt;fundamental_rules&gt;<br>    1. Assess clarity first. If a request lacks detail, ask for clarification.  <br>    2. Validate API availability. If an API is unavailable, inform the user.  <br>    3. Do NOT assume the time frame—ask the user to provide the start time, ensuring the maximum allowable range is the last 3 hours.<br>    4. Convert Unix timestamps to human-readable format.  <br>    5. Optimize execution by making parallel API calls.  <br>    7. Never infer missing inputs—always request details explicitly.  <br>  &lt;/fundamental_rules&gt;<br>&lt;/core_guidelines&gt;<br><br>&lt;response_strategy&gt;<br>  &lt;initial_assessment&gt;<br>    1. Rephrase the request to confirm understanding.  <br>    2. Identify missing details and ask for clarification.  <br>  &lt;/initial_assessment&gt;<br><br>  &lt;execution_and_validation&gt;<br>    3. Query APIs simultaneously when possible.  <br>    4. Cross-check responses for consistency.  <br>    5. Validate outputs for accuracy and completeness.  <br>  &lt;/execution_and_validation&gt;<br><br>  &lt;error_handling&gt;<br>    6. If an API or function fails, report the exact error immediately.  <br>    7. Never generate or assume missing information.  <br>  &lt;/error_handling&gt;<br><br>  &lt;output_presentation&gt;<br>    8. Ensure responses are clear, structured, and concise.  <br>    9. Format timestamps properly for readability.  <br>    10. Prioritize important details in responses.  <br>  &lt;/output_presentation&gt;<br>&lt;/response_strategy&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9eO-Cc-2s-qQ6mujGJ4vQ.png" /></figure><p>Leave <strong>Additional Settings</strong> as default. Click <strong>Save</strong> in the top-right corner of the screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZPN3jSiV7TOJKtiX6ae4Q.png" /></figure><p>5 . Scroll down and click <strong>Add</strong> under <strong>Action Groups</strong></p><ul><li><strong>Action group name</strong>metrics-action-group</li><li><strong>Description</strong>Queries CloudWatch Metrics Insights to retrieve and monitor AWS service performance.</li><li><strong>Action group type: Define with API schemasSelect how to define the Lambda function</strong></li><li><strong>Select an existing Lambda function: Select Lambda functionMetricQueryHandler</strong></li><li><strong>Action group schema: Select an existing API schema</strong></li><li><strong>S3 Url: </strong>Click <strong>Browse S3</strong>, navigate to <strong>codepipeline-assetsourcebucket-xxxxx</strong>, then open <strong>assets/</strong> folder and select <strong>MetricOpenAPI.yml</strong> and click Choose</li></ul><h3>Agent Operations</h3><p>Once your Bedrock Agent is created, you’ll focus on the core elements that define how it operates and responds to monitoring queries. In this section you’ll complete the following:</p><ul><li><strong>Agent Instructions:</strong> You’ll review the rules and guidelines that shape your agent’s behavior, from how it handles incoming queries to how it formats responses and manages errors.</li><li><strong>Action Groups:</strong> You’ll explore how action groups connect your agent to external systems, like CloudWatch. These groups define the specific API actions your agent can invoke to retrieve real-time performance data.</li><li><strong>Testing Your Agent:</strong> You’ll interact directly with your agent using sample queries, watching how it applies its instructions and action groups to answer your questions using live CloudWatch data.</li><li><strong>Analyzing Traces:</strong> You’ll review detailed traces to see how your agent breaks down each query into decisions and actions. Traces help you understand how the agent interprets requests, selects actions, and handles errors — providing valuable insight for debugging and optimization.</li></ul><p>Together, these steps give you full visibility into how your agent operates, from initial request to final response — so you can confidently extend and refine its capabilities.</p><h3>Agent Instructions</h3><p>Agent instructions provide the <strong>rules and guidelines</strong> for how your agent handles monitoring requests, formats responses, and manages errors or missing details.</p><h4>Why Structure Matters</h4><p>To ensure the foundation model follows these rules correctly, we use <strong>structured XML tags</strong> to organize the instructions. Clear labeling helps the agent distinguish between operational rules and actual monitoring data. Without this structure, the model could confuse instructions with query data — especially in complex workflows. These tags serve two key purposes:</p><ul><li>&lt;core_guidelines&gt;</li><li>&lt;response_strategy&gt;</li></ul><p>Core Guidelines</p><p>The first part of your agent’s instruction defines its core role, responsibilities, and rules. This section sets the agent’s purpose, tone, and how it handles user requests and errors.</p><pre>&lt;core_guidelines&gt;<br>You are an AI-powered monitoring assistant ensuring clarity, accuracy, and real-time insights. <br>  &lt;fundamental_rules&gt;<br>    1. Assess clarity first. If a request lacks detail, ask for clarification.  <br>    2. Validate API availability. If an API is unavailable, inform the user.  <br>    3. Do NOT assume the time frame—ask the user to provide the start time, ensuring the maximum allowable range is the last 3 hours. <br>    4. Convert Unix timestamps to human-readable format.  <br>    5. Optimize execution by making parallel API calls.  <br>    6. Handle errors strictly. If a function fails, stop execution and report the exact error message.  <br>    7. Never infer missing inputs—always request details explicitly.  <br>  &lt;/fundamental_rules&gt;<br>&lt;/core_guidelines&gt;</pre><p>This ensures the agent begins every interaction with clear rules on <strong>what to do, what to avoid, and how to respond.</strong></p><p>Response Strategy</p><p>The <strong>response strategy</strong> defines how the agent processes each request — from understanding the initial input to validating data, handling errors, and formatting the final response.</p><pre>&lt;response_strategy&gt;<br>  &lt;initial_assessment&gt;<br>    1. Rephrase the request to confirm understanding.  <br>    2. Identify missing details and ask for clarification.  <br>  &lt;/initial_assessment&gt;<br><br>  &lt;execution_and_validation&gt;<br>    3. Query APIs simultaneously when possible.  <br>    4. Validate outputs for accuracy and completeness.  <br>  &lt;/execution_and_validation&gt;<br><br>  &lt;error_handling&gt;<br>    5. If an API or function fails, report the exact error immediately.  <br>    6. Never generate or assume missing information.  <br>  &lt;/error_handling&gt;<br><br>  &lt;output_presentation&gt;<br>    7. Ensure responses are clear, structured, and concise.  <br>    8. Format timestamps properly for readability.  <br>  &lt;/output_presentation&gt;<br>&lt;/response_strategy&gt;</pre><h3>Agent Action Groups</h3><p>Action groups define the <strong>specific tasks your agent can perform</strong>. In this monitoring solution, the metrics-action-group allows your agent to query CloudWatch Metrics Insights to retrieve performance data.</p><p>This action group includes:</p><ul><li><strong>getDimensionMetricDataAndNamespace</strong> — Identify the AWS service, dimensions, and metrics relevant to the user’s question.</li><li><strong>getMetricQueryString</strong> — Build a CloudWatch Metric Insights query using the identified namespace, dimension, and metrics.</li><li><strong>executeMetricQuery</strong> — Execute the query to retrieve monitoring data.</li><li><strong>getCurrentTime</strong> — Provide the current Unix timestamp to define query time ranges.</li></ul><p>Each action is defined as an <strong>API operation</strong> with specific parameters and response formats, allowing the agent to make <strong>precise, validated API calls</strong>.</p><h4>OpenAPI Schema Structure</h4><p>Action groups in Amazon Bedrock agents can use <strong>OpenAPI schemas</strong> to define available actions. Each schema acts as a <strong>contract</strong> between the agent and the underlying Lambda function. It defines:</p><ul><li><strong>Available operations (actions)</strong></li><li><strong>Required parameters</strong></li><li><strong>Response formats</strong></li><li><strong>Descriptions to help the agent select the correct action</strong></li></ul><p>This ensures the agent always <strong>sends the correct inputs</strong> and <strong>expects the correct outputs</strong>, allowing seamless interaction with your Lambda function, which handles the actual call to CloudWatch or other AWS services.</p><h4>Example Action Breakdown</h4><p>Here’s how the <strong>getDimensionMetricDataAndNamespace</strong> action is defined in the OpenAPI schema:</p><h4>OpenAPI Schema Structure</h4><p>Action groups in Amazon Bedrock agents can use <strong>OpenAPI schemas</strong> to define available actions. Each schema acts as a <strong>contract</strong> between the agent and the underlying Lambda function. It defines:</p><ul><li><strong>Available operations (actions)</strong></li><li><strong>Required parameters</strong></li><li><strong>Response formats</strong></li><li><strong>Descriptions to help the agent select the correct action</strong></li></ul><p>This ensures the agent always <strong>sends the correct inputs</strong> and <strong>expects the correct outputs</strong>, allowing seamless interaction with your Lambda function, which handles the actual call to CloudWatch or other AWS services.</p><h4>Example Action Breakdown</h4><p>Here’s how the <strong>getDimensionMetricDataAndNamespace</strong> action is defined in the OpenAPI schema:</p><pre>paths:<br>  /getDimensionMetricDataAndNamespace:<br>    post:<br>      operationId: &quot;getDimensionMetricDataAndNamespace&quot;<br>      description: &quot;Identify the AWS service, dimensions, and metrics relevant to the user&#39;s query.&quot;<br>      requestBody:<br>        required: true<br>        content:<br>          application/json:<br>            schema:<br>              type: object<br>              properties:<br>                question:<br>                  type: string<br>                  description: &quot;The user’s question providing context for the dimension and metric selection.&quot;<br>              required:<br>                - question<br>      responses:<br>        &quot;200&quot;:<br>          description: &quot;Dimension and metric data retrieval successful.&quot;<br>          content:<br>            application/json:<br>              schema:<br>                type: object<br>                properties:<br>                  dimension_metric_data:<br>                    type: string<br>                    description: &quot;The relevant dimension, metric, and namespace data based on the user&#39;s question.&quot;<br>                  namespace:<br>                    type: string<br>                    description: &quot;The relevant namespace data based on the user&#39;s question.&quot;</pre><h4>Action Summary</h4><ul><li><strong>Name</strong>getDimensionMetricDataAndNamespace</li><li><strong>Expected Parameter</strong>question – A string describing the user’s query (e.g., &quot;What is the average CPU utilization for my EC2 instances?&quot;)</li><li><strong>Response Properties</strong>- dimension_metric_data – Dimension and metric information (e.g., Dimension set: (&#39;InstanceId&#39;), Metric: CPUUtilization). - namespace – The AWS namespace associated with the service (e.g., AWS/EC2).</li><li><strong>Purpose</strong>Identify the AWS service, dimensions, and metrics relevant to the user&#39;s query.</li></ul><p>This structured approach ensures that:</p><ul><li>The agent knows exactly what’s required for each action, so when it invokes lambda it sends the correct parameters and input.</li><li>The Lambda function knows how to parse incoming requests and return structured responses.</li><li>Adding new actions or updating parameters can be done in a <strong>centralized OpenAPI file</strong>, without changing the agent’s core logic.</li></ul><h4>Role of the Lambda Function</h4><p>The agent itself does not directly call CloudWatch or other AWS services. Instead, the agent sends requests to a MetricQueryHandler Lambda function, which acts as a bridge between the agent and Amazon Cloudwatch.</p><p>This design <strong>decouples</strong> the agent from direct API interactions, allowing you to:</p><ul><li>Centralize all logic, error handling, and caching within the Lambda.</li><li>Update or extend functionality (e.g., adding support for new services) without modifying the agent itself.</li></ul><p>When the agent invokes the getDimensionMetricDataAndNamespace action, it sends a request to the <strong>MetricQueryHandler</strong> Lambda function with a payload like this:</p><pre>{<br>    &quot;messageVersion&quot;: &quot;1.0&quot;,<br>    &quot;parameters&quot;: [],<br>    &quot;sessionId&quot;: &quot;989978684154843&quot;,<br>    &quot;actionGroup&quot;: &quot;metrics-action-group&quot;,<br>    &quot;agent&quot;: {<br>        &quot;name&quot;: &quot;petwatch-agent&quot;,<br>        &quot;version&quot;: &quot;DRAFT&quot;,<br>        &quot;id&quot;: &quot;GBKQH4FOPA&quot;,<br>        &quot;alias&quot;: &quot;TSTALIASID&quot;<br>    },<br>    &quot;sessionAttributes&quot;: {},<br>    &quot;promptSessionAttributes&quot;: {},<br>    &quot;inputText&quot;: &quot;the last 30 minutes&quot;,<br>    &quot;httpMethod&quot;: &quot;POST&quot;,<br>    &quot;apiPath&quot;: &quot;/getDimensionMetricDataAndNamespace&quot;,<br>    &quot;requestBody&quot;: {<br>        &quot;content&quot;: {<br>            &quot;application/json&quot;: {<br>                &quot;properties&quot;: [<br>                    {<br>                        &quot;name&quot;: &quot;question&quot;,<br>                        &quot;type&quot;: &quot;string&quot;,<br>                        &quot;value&quot;: &quot;What is the average CPU utilization for my EC2 instances?&quot;<br>                    }<br>                ]<br>            }<br>        }<br>    }<br>}</pre><p>This separation of responsibilities keeps the agent focused on orchestration and decision-making, while the Lambda function handles service-specific logic, error handling, and external API calls.</p><h4>Sample Lambda Handler</h4><p>The Lambda function processes the request, extracts the user’s question, identifies the <strong>most relevant namespace</strong>, retrieves <strong>dimension and metric data</strong> from CloudWatch, and uses a <strong>Bedrock LLM</strong> to select the best metric/dimension combination. Finally, it returns the results to the agent.</p><pre>def get_dimension_metric_data(question):<br>    namespace_result = get_namespace(question)<br>    if namespace_result.get(&quot;error&quot;):<br>        return {&quot;error&quot;: namespace_result[&quot;error&quot;]}<br><br>    namespace = namespace_result[&quot;namespace&quot;]<br>    metrics_result = get_cached_dimension_metrics(namespace)<br>    if metrics_result.get(&quot;error&quot;):<br>        return {&quot;namespace&quot;: namespace, &quot;error&quot;: metrics_result[&quot;error&quot;]}<br><br>    formatted_metrics = format_dimension_combinations(metrics_result)<br><br>    llm_prompt = f&quot;&quot;&quot;<br>    Based on the following question: &#39;{question}&#39;, and these available metrics:<br>    {formatted_metrics}<br>    Identify the most relevant dimension set and metric for answering the question.<br>    &quot;&quot;&quot;<br><br>    llm_response = invoke_llm(llm_prompt)<br><br>    return {<br>        &quot;dimension_metric_data&quot;: llm_response[&quot;result&quot;],<br>        &quot;namespace&quot;: namespace,<br>    }<br><br><br>def lambda_handler(event, context):<br>    if event.get(&quot;apiPath&quot;) == &quot;/getDimensionMetricDataAndNamespace&quot; and event.get(&quot;httpMethod&quot;) == &quot;POST&quot;:<br>        question = extract_parameter(event, &quot;question&quot;)<br>        result = get_dimension_metric_data(question)<br>        return {<br>            &quot;messageVersion&quot;: &quot;1.0&quot;,<br>            &quot;response&quot;: {<br>                &quot;actionGroup&quot;: event.get(&quot;actionGroup&quot;),<br>                &quot;apiPath&quot;: event.get(&quot;apiPath&quot;),<br>                &quot;httpMethod&quot;: event.get(&quot;httpMethod&quot;),<br>                &quot;httpStatusCode&quot;: 200,<br>                &quot;responseBody&quot;: {<br>                    &quot;application/json&quot;: {<br>                        &quot;body&quot;: json.dumps({&quot;result&quot;: result})<br>                    }<br>                }<br>            }<br>        }<br><br></pre><h4>Workflow Example</h4><p>To see how these actions work together, let’s walk through how your agent processes the following query:</p><p><strong>User Prompt:</strong></p><pre>What is the average CPU utilization of my EC2 instances for the last 30 minutes, listed by instance ID?</pre><h4>Step 1: Determine Time Range</h4><p>To define the time range, the agent calls getCurrentTime to retrieve the current Unix timestamp and today&#39;s date in Eastern Time.</p><p><strong>Example Response:</strong></p><pre>{<br>  &quot;result&quot;: {<br>    &quot;current_unix_timestamp&quot;: 1740696674,<br>    &quot;todays_date_in_eastern_time&quot;: &quot;2025-02-27 05:51:14 PM&quot;<br>  }<br>}</pre><h4>Step 2: Retrieve Dimensions, Metric &amp; Namespace</h4><p>The agent calls getDimensionMetricDataAndNamespace, which identifies the service (AWS/EC2), the metric (CPUUtilization), and the relevant dimension (InstanceId).</p><p><strong>Example response:</strong></p><pre>{<br>  &quot;result&quot;: {<br>    &quot;namespace&quot;: &quot;AWS/EC2&quot;,<br>    &quot;dimension_metric_data&quot;: &quot;Dimension: InstanceId, Metric: CPUUtilization&quot;<br>  }<br>}</pre><h4>Step 3: Generate the Query String</h4><p>Next, the agent invokes getMetricQueryString, passing the <strong>namespace</strong>, <strong>dimension</strong>, <strong>metric</strong>, and the user’s <strong>question</strong> to generate a CloudWatch Metrics Insights query string. <strong>Example response:</strong></p><pre>{<br>    &quot;result&quot;: &quot;SELECT AVG(&quot;CPUUtilization&quot;) FROM SCHEMA(&quot;AWS/EC2&quot;, InstanceId)&quot;<br>}</pre><h4>Step 4: Execute the Query</h4><p>Using the <strong>current Unix timestamp</strong>, the agent calculates the <strong>start time</strong> and <strong>end time</strong> for the query. It then sends the <strong>query string</strong>, along with the calculated time range, to <strong>executeMetricQuery</strong>, which retrieves the CloudWatch data.</p><p><strong>Example response:</strong></p><pre>{<br>  &quot;result&quot;: {<br>    &quot;MetricDataResults&quot;: [<br>      {<br>        &quot;Id&quot;: &quot;e1&quot;,<br>        &quot;Label&quot;: &quot;Instance i-03a79c41aa3dedc77&quot;,<br>        &quot;Timestamps&quot;: [<br>          &quot;2024-09-14 02:40:00+00:00&quot;,<br>          &quot;2024-09-14 02:35:00+00:00&quot;,<br>          &quot;2024-09-14 02:30:00+00:00&quot;,<br>          &quot;2024-09-14 02:25:00+00:00&quot;,<br>          &quot;2024-09-14 02:20:00+00:00&quot;,<br>          &quot;2024-09-14 02:15:00+00:00&quot;<br>        ],<br>        &quot;Values&quot;: [0.82, 0.85, 0.84, 0.83, 0.82, 0.83],<br>        &quot;StatusCode&quot;: &quot;Complete&quot;<br>      },<br>      {<br>        &quot;Id&quot;: &quot;e1&quot;,<br>        &quot;Label&quot;: &quot;Instance i-03e81d3e45cc62e4a&quot;,<br>        &quot;Timestamps&quot;: [<br>          &quot;2024-09-14 02:40:00+00:00&quot;,<br>          &quot;2024-09-14 02:35:00+00:00&quot;,<br>          &quot;2024-09-14 02:30:00+00:00&quot;,<br>          &quot;2024-09-14 02:25:00+00:00&quot;,<br>          &quot;2024-09-14 02:20:00+00:00&quot;,<br>          &quot;2024-09-14 02:15:00+00:00&quot;<br>        ],<br>        &quot;Values&quot;: [4.07, 4.06, 3.74, 3.88, 4.27, 4.3],<br>        &quot;StatusCode&quot;: &quot;Complete&quot;<br>      }<br>    ]<br>  }<br>}</pre><h4>Query 1: Load Balancer Requests</h4><p>Try this query to get started by copying it, pasting it into the query text box and clicking <strong>Run</strong>:</p><pre>What is the request count for each application load balancer? List each load balancer by name along with its respective Count.</pre><p>If you see an <strong>Access Denied when calling Bedrock</strong> error, verify you’ve completed the <a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/20prerequisites/21-bedrock-setup/"><strong>Enable Model Access</strong></a> setup in the prerequisites section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/0*KE7O0Jp6Vl90LdGe" /></figure><p>If the agent prompts for a time frame, you can use natural language phrases like:</p><pre>“the last 30 minutes”<br>“between 8 and 9 AM Eastern Time”</pre><p>Note: Please specify all times in Eastern Time, and ensure they fall within the last three hours from the current time.</p><p>IMPORTANT</p><ul><li>CloudWatch queries are limited to the most recent 3 hours of data</li><li>The agent currently understands Eastern Time zone</li><li>Responses typically take 15–30 seconds as multiple API calls are coordinated</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*Cbij_rw3fKHTAcoj" /></figure><p>You should have a list of Load Balancers, along with their corresponding <strong>request count</strong> values. These are the Application Load Balancers current running in your account.</p><h4>Query 2: Availability Zone Analysis</h4><p>Try this more detailed query:</p><pre>What is the request count for each application load balancer, per availability zone? List each load balancer along with its respective Count for each availability zone.</pre><p>Once you click <strong>Run</strong>, the agent will return data filtered by <strong>Availability Zone.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/0*gaZXPwWYLIOZpgy3" /></figure><h3>Analyzing Traces</h3><p>Traces show how the Bedrock agent processes a user query by breaking it down into individual steps. Traces are extremely helpful during development for quickly debugging and understanding the agent’s workflow.</p><h4>Exploring Traces</h4><p><strong>Step 1: Start a New Session</strong><br>Refresh your browser to ensure you’re starting with a clean trace log or use the “broom” icon at top of the Test panel to end the current session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B4JJylI3Ci80eRWa" /></figure><p><strong>Step 2: Run a Sample Query</strong><br>Copy and paste this query and click <strong>Run</strong> to see traces in action.</p><pre>What is the average CPU utilization of my EC2 instances for the last 30 minutes, listed by ID?</pre><p><strong>Step 3: Access the Trace</strong><br>While the agent processes your request click <strong>Show trace</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/0*6gOjYnDZTOrcFEUr" /></figure><h4>Understanding Trace Components</h4><p>Let’s explore a trace step’s object. Expand <strong>Trace Step 1</strong> and examine the content of each of the following keys:</p><p><strong>modelInvocationInput</strong> — This section contains details about:</p><ul><li>Initial configuration</li><li>Context information</li><li>Agent instructions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sjhYkNcgRQgLb3l6" /></figure><p>The text field contains crucial context from the agent&#39;s instructions and action groups that we had explored in the previous steps.</p><p><strong>rationale</strong> — This section reveals the agent’s thought process:</p><ul><li>Analysis of the query</li><li>Planning the next actions</li><li>Decision-making logic</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bBlBRNUuI3pscf9o" /></figure><p><strong>invocationInput</strong> — Shows the specific API calls the agent makes, including:</p><ul><li>Selected endpoint</li><li>Parameters</li><li>Request body details</li></ul><p>In the image below, the agent is calling the /getDimensionMetricDataAndNamespace endpoint from the metrics-action-group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cWh5yLpFc1s7HKoV" /></figure><p><strong>observation</strong> — Displays the results of API calls:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gCjBTMSULoJewm82" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47a1d8dfe8c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Date Warehouse with Amazon Redshift]]></title>
            <link>https://medium.com/@tdhtp2016/date-warehouse-with-amazon-redshift-b9e0550002cd?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9e0550002cd</guid>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 09:08:49 GMT</pubDate>
            <atom:updated>2025-04-23T09:08:49.053Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*akdmKIe_3WPNsdMP" /></figure><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#create-tables">Create Tables</a></h3><p>Amazon Redshift is an ANSI SQL compliant data warehouse. You can create tables using familiar <a href="https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html">CREATE TABLE </a>statements.</p><p>To create 8 tables from TPC Benchmark data model, copy the following create table statements and run them. Given below is the data model for the tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*uWaAHBee6wT-7BGz" /></figure><pre><br>DROP TABLE IF EXISTS partsupp;<br>DROP TABLE IF EXISTS lineitem;<br>DROP TABLE IF EXISTS supplier;<br>DROP TABLE IF EXISTS part;<br>DROP TABLE IF EXISTS orders;<br>DROP TABLE IF EXISTS customer;<br>DROP TABLE IF EXISTS nation;<br>DROP TABLE IF EXISTS region;<br><br>CREATE TABLE region (<br>  R_REGIONKEY bigint NOT NULL,<br>  R_NAME varchar(25),<br>  R_COMMENT varchar(152))<br>diststyle all;<br>CREATE TABLE nation (<br>  N_NATIONKEY bigint NOT NULL,<br>  N_NAME varchar(25),<br>  N_REGIONKEY bigint,<br>  N_COMMENT varchar(152))<br>diststyle all;<br>create table customer (<br>  C_CUSTKEY bigint NOT NULL,<br>  C_NAME varchar(25),<br>  C_ADDRESS varchar(40),<br>  C_NATIONKEY bigint,<br>  C_PHONE varchar(15),<br>  C_ACCTBAL decimal(18,4),<br>  C_MKTSEGMENT varchar(10),<br>  C_COMMENT varchar(117))<br>diststyle all;<br>create table orders (<br>  O_ORDERKEY bigint NOT NULL,<br>  O_CUSTKEY bigint,<br>  O_ORDERSTATUS varchar(1),<br>  O_TOTALPRICE decimal(18,4),<br>  O_ORDERDATE Date,<br>  O_ORDERPRIORITY varchar(15),<br>  O_CLERK varchar(15),<br>  O_SHIPPRIORITY Integer,<br>  O_COMMENT varchar(79))<br>distkey (O_ORDERKEY)<br>sortkey (O_ORDERDATE);<br>create table part (<br>  P_PARTKEY bigint NOT NULL,<br>  P_NAME varchar(55),<br>  P_MFGR  varchar(25),<br>  P_BRAND varchar(10),<br>  P_TYPE varchar(25),<br>  P_SIZE integer,<br>  P_CONTAINER varchar(10),<br>  P_RETAILPRICE decimal(18,4),<br>  P_COMMENT varchar(23))<br>diststyle all;<br>create table supplier (<br>  S_SUPPKEY bigint NOT NULL,<br>  S_NAME varchar(25),<br>  S_ADDRESS varchar(40),<br>  S_NATIONKEY bigint,<br>  S_PHONE varchar(15),<br>  S_ACCTBAL decimal(18,4),<br>  S_COMMENT varchar(101))<br>diststyle all;<br>create table lineitem (<br>  L_ORDERKEY bigint NOT NULL,<br>  L_PARTKEY bigint,<br>  L_SUPPKEY bigint,<br>  L_LINENUMBER integer NOT NULL,<br>  L_QUANTITY decimal(18,4),<br>  L_EXTENDEDPRICE decimal(18,4),<br>  L_DISCOUNT decimal(18,4),<br>  L_TAX decimal(18,4),<br>  L_RETURNFLAG varchar(1),<br>  L_LINESTATUS varchar(1),<br>  L_SHIPDATE date,<br>  L_COMMITDATE date,<br>  L_RECEIPTDATE date,<br>  L_SHIPINSTRUCT varchar(25),<br>  L_SHIPMODE varchar(10),<br>  L_COMMENT varchar(44))<br>distkey (L_ORDERKEY)<br>sortkey (L_RECEIPTDATE);<br>create table partsupp (<br>  PS_PARTKEY bigint NOT NULL,<br>  PS_SUPPKEY bigint NOT NULL,<br>  PS_AVAILQTY integer,<br>  PS_SUPPLYCOST decimal(18,4),<br>  PS_COMMENT varchar(199))<br>diststyle even;</pre><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#data-loading">Data Loading</a></h3><p>In this lab, you will learn following approaches to load data into Redshift.</p><ol><li>Load data from S3 to Redshift. You will use this process for loading 7 tables out of 8 tables created in above step.</li><li>Load data from your desktop file to Redshift. You will use this process for loading 1 table out of 8 created in above step i.e. nation table.</li></ol><p>We split the load process to make you familiar with both the approaches.</p><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#loading-data-from-s3">Loading data from S3</a></h3><p><a href="https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html">COPY command </a>loads large amounts of data effectively from Amazon S3 into Amazon Redshift. A single COPY command can load from multiple files into one table. It automatically loads the data in parallel from all the files present in the S3 location you provide.</p><p>If the target table is empty, COPY Command can perform compression encoding for each column automatically based on the column datatype so that you need not worry about choosing the right compression algorithm for each column. To ensure that Redshift performs a compression analysis, we are going to set the COMPUPDATE parameter to PRESET in the COPY commands.</p><p>The sample data to load is made available in a public <a href="https://s3.console.aws.amazon.com/s3/buckets/redshift-immersionday-labs?region=us-west-2&amp;prefix=data/&amp;showversions=false">Amazon S3 bucket </a>.</p><p>Copy the following statements to load data into 7 tables.</p><pre>COPY region FROM &#39;s3://redshift-immersionday-labs/data/region/region.tbl.lzo&#39;<br>iam_role default<br>region &#39;us-west-2&#39; lzop delimiter &#39;|&#39; COMPUPDATE PRESET;<br><br>copy customer from &#39;s3://redshift-immersionday-labs/data/customer/customer.tbl.&#39;<br>iam_role default<br>region &#39;us-west-2&#39; lzop delimiter &#39;|&#39; COMPUPDATE PRESET;<br>copy orders from &#39;s3://redshift-immersionday-labs/data/orders/orders.tbl.&#39;<br>iam_role default<br>region &#39;us-west-2&#39; lzop delimiter &#39;|&#39; COMPUPDATE PRESET;<br>copy part from &#39;s3://redshift-immersionday-labs/data/part/part.tbl.&#39;<br>iam_role default<br>region &#39;us-west-2&#39; lzop delimiter &#39;|&#39; COMPUPDATE PRESET;<br>copy supplier from &#39;s3://redshift-immersionday-labs/data/supplier/supplier.json&#39; manifest<br>iam_role default<br>region &#39;us-west-2&#39; lzop delimiter &#39;|&#39; COMPUPDATE PRESET;<br>copy lineitem from &#39;s3://redshift-immersionday-labs/data/lineitem-part/&#39;<br>iam_role default<br>region &#39;us-west-2&#39; gzip delimiter &#39;|&#39; COMPUPDATE PRESET;<br>copy partsupp from &#39;s3://redshift-immersionday-labs/data/partsupp/partsupp.tbl.&#39;<br>iam_role default<br>region &#39;us-west-2&#39; lzop delimiter &#39;|&#39; COMPUPDATE PRESET;</pre><p>The estimated time to load the data is as follows. While waiting for the copy process to finish, you can move to next section <a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#loading-data-from-your-local-machine-using-qev2">Loading data from your local machine using QEV2</a></p><ul><li>REGION (5 rows) — 2s</li><li>CUSTOMER (15M rows) — 2m</li><li>ORDERS — (76M rows) — 10s</li><li>PART — (20M rows) — 2m</li><li>SUPPLIER — (1M rows) — 10s</li><li>LINEITEM — (303M rows) — 22s</li><li>PARTSUPPLIER — (80M rows) — 15s</li></ul><blockquote>Note: A few key takeaways from the above COPY statements.</blockquote><ol><li>COMPUPDATE PRESET ON will assign compression using the Amazon Redshift best practices related to the data type of the column but without analyzing the data in the table.</li><li>COPY for the REGION table points to a specific file (region.tbl.lzo) while COPY for other tables point to a prefix to multiple files (lineitem.tbl.).</li><li>COPY for the SUPPLIER table points a manifest file (supplier.json). A manifest file is a JSON file that lists the files to be loaded and their locations.</li></ol><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#loading-data-from-your-local-machine-using-qev2">Loading data from your local machine using QEV2</a></h3><p>You will be loading the nation table through query editor v2 by importing the data from your local machine.</p><p><strong><em>One time setup</em></strong></p><p>As a one time setup, you need to create a S3 bucket in the same region as redshift instance. And add the S3 bucket path in QEV2 account setting page.</p><p>S3 bucket is already pre-created in this lab environment. Navigate to <a href="https://console.aws.amazon.com/cloudformation/home">AWS CloudFormation </a>and click on ‘cfn’ stack. Under ‘Outputs’ tab note down the S3Bucket. You’ll need this value in next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nodAENbznhD5B_4C" /></figure><p>In QEV2 go to <strong>Settings</strong>(bottom left) — <strong>Account settings</strong> — <strong>S3 bucket name</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/0*vY8NSYOQ-iaPMaLG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*bf8Ab1rxSz0Q1vpy" /></figure><p><strong><em>Load File</em></strong></p><p>You can download the sample Nation.csv file onto your local machine using the below link.</p><p><a href="https://redshift-immersionday-labs.s3.us-west-2.amazonaws.com/data/nation/Nation.csv">Nation fileNation file</a>Nation file</p><p>Let us load the downloaded nation file on your desktop into Redshift.</p><ol><li>From resources section, select and connect to your serverless workgroup.</li></ol><p>The <em>Nation</em> table is already created in the <a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#create-tables">Create Tables</a> step. However you can also create a table through user interface by following <strong>Create</strong> →<strong>Table</strong> option in left navigation pane in Query Editor V2</p><ol><li>Click on <strong>Load data</strong> — select <strong>Load from local file</strong> — <strong>Browse</strong> and select the <em>Nation.csv</em> file downloaded earlier — <strong>Data conversion parameters</strong> — <strong>Check Ignore header rows</strong> — <strong>Next</strong> — <strong>Table options</strong> select your <em>serverless work group</em> — <strong>Database</strong> = <em>dev</em> — <strong>Schema</strong> = <em>public</em> — <strong>Table</strong> = <em>nation</em> — <strong>Load data</strong> — Notification of <strong><em>Loading data from a local file succeeded</em></strong> would be shown on top.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*4029JrRS1LNK6EC_" /></figure><p>Machines with upload restrictions</p><p>If your machine has restrictions that don’t allow uploads to S3 you will not be able to load a local file. However, you can use the Query Editor v2 to load a file directly from S3. To do this, select the <strong>Load from S3 bucket</strong> option and input the following location:</p><p>s3://redshift-immersionday-labs/data/nation/Nation.csv</p><p>You will need to select the S3 location as <strong>us-west-2</strong>.</p><p>Follow the below instructions as-is. The only change is selecting the IAM role when choosing which table to load. There will be only one role available in AWS Sponsered Events (RedshiftServerlessImmersionRole).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ew4S2I0iyNYKbbob" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BVTfDfIVYpOBlDFC" /></figure><p>You can also manually load a file in S3 following the same approach. You just need to select “Load from S3” instead of “Load from local file”. Please refer <a href="https://www.youtube.com/watch?v=Mrx5fG6TbHM">https://www.youtube.com/watch?v=Mrx5fG6TbHM </a>for reference.</p><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#load-validation">Load Validation</a></h3><p>Let us do a quick check of counts on few tables to ensure data is loaded as expected. You can open a new editor and run below queries if the copy process is still loading some tables.</p><pre>--Number of rows= 5<br>select count(*) from region;<br>--Number of rows= 25<br>select count(*) from nation;<br>--Number of rows= 76,000,000<br>select count(*) from orders;</pre><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#troubleshooting-loads">Troubleshooting Loads</a></h3><p>To troubleshoot any data load issues, you can query SYS_LOAD_ERROR_DETAIL.</p><p>In addition, you can validate your data without actually loading the table. You can use the NOLOAD option with the COPY command to make sure that your data will load without any errors before running the actual data load. Notice that running COPY with the NOLOAD option is much faster than loading the data since it only parses the files.</p><p>Let’s try to load the CUSTOMER table with a different data file with mismatched columns.</p><pre>COPY customer FROM &#39;s3://redshift-immersionday-labs/data/nation/nation.tbl.&#39;<br>iam_role default<br>region &#39;us-west-2&#39; lzop delimiter &#39;|&#39; noload;</pre><p>You will get the following error.</p><pre>ERROR: Load into table &#39;customer&#39; failed. Check &#39;sys_load_error_detail&#39; system table for details.</pre><p>Query the STL_LOAD_ERROR system table for details.</p><pre>select * from SYS_LOAD_ERROR_DETAIL;</pre><p>Notice that there is one row for the column <strong>c_nationkey</strong> with datatype <strong>int</strong> with an error message “Invalid digit, Value ‘h’, Pos 1” indicating that you are trying to load character into an integer column.</p><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/lab2#automatic-table-maintenance-analyze-and-vacuum">Automatic Table Maintenance — ANALYZE and VACUUM</a></h3><p><strong>Analyze:</strong></p><p>When loading into an empty table, the COPY command by default collects statistics (ANALYZE). If you are loading a non-empty table using COPY command, in most cases, you don’t need to explicitly run the ANALYZE command. Amazon Redshift monitors changes to your workload and automatically updates statistics in the background. To minimize impact to your system performance, automatic analyze runs during periods when workloads are light.</p><p>If you need to analyze the table immediately after load, you can still manually run ANALYZE command.</p><p>To run ANALYZE on <strong>orders</strong> table, copy the following command and run it.</p><pre>ANALYZE orders;</pre><p><strong>Vacuum:</strong></p><p><em>Vacuum Delete:</em> When you perform delete on a table, the rows are marked for deletion(soft deletion), but are not removed. When you perform an update, the existing rows are marked for deletion(soft deletion) and updated rows are inserted as new rows. Amazon Redshift automatically runs a VACUUM DELETE operation in the background to reclaim disk space occupied by rows that were marked for deletion by UPDATE and DELETE operations, and compacts the table to free up the consumed space. To minimize impact to your system performance, automatic VACUUM DELETE runs during periods when workloads are light.</p><p>If you need to reclaim diskspace immediately after a large delete operation, for example after a large data load, then you can still manually run the VACUUM DELETE command. Lets see how VACUUM DELETE reclaims table space after delete operation.</p><p>First, capture <strong>tbl_rows</strong>(Total number of rows in the table. This value includes rows marked for deletion, but not yet vacuumed) and <strong>estimated_visible_rows</strong>(The estimated visible rows in the table. This value does not include rows marked for deletion) for the ORDERS table. Copy the following command and run it.</p><pre>select &quot;table&quot;, size, tbl_rows, estimated_visible_rows<br>from SVV_TABLE_INFO<br>where &quot;table&quot; = &#39;orders&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*zyozH8053i8Mf0YISOD8kA.png" /></figure><p>Next, delete rows from the ORDERS table. Copy the following command and run it.</p><pre>delete orders where o_orderdate between &#39;1997-01-01&#39; and &#39;1998-01-01&#39;;</pre><p>Next, capture the tbl_rows and estimated_visible_rows for ORDERS table after the deletion.</p><p>Copy the following command and run it. Notice that the tbl_rows value hasn’t changed, after deletion. This is because rows are marked for soft deletion, but VACUUM DELETE is not yet run to reclaim space.</p><pre>select &quot;table&quot;, size, tbl_rows, estimated_visible_rows<br>from SVV_TABLE_INFO<br>where &quot;table&quot; = &#39;orders&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*jI3FfoqhwEsLU67jtWJYWg.png" /></figure><p>Now, run the VACUUM DELETE command. Copy the following command and run it.</p><pre>vacuum delete only orders;</pre><p>Confirm that the VACUUM command reclaimed space by running the following query again and noting the tbl_rows value has changed.</p><pre>select &quot;table&quot;, size, tbl_rows, estimated_visible_rows<br>from SVV_TABLE_INFO<br>where &quot;table&quot; = &#39;orders&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*fpxQ1XB01uMBFqbIOgpANQ.png" /></figure><p><em>Vacuum Sort:</em> When you define <a href="https://docs.aws.amazon.com/redshift/latest/dg/t_Sorting_data.html">SORT KEYS </a>on your tables, Amazon Redshift automatically sorts data in the background to maintain table data in the order of its sort key. Having sort keys on frequently filtered columns makes block level pruning, which is already efficient in Amazon Redshift, more efficient.</p><p>Amazon Redshift keeps track of your scan queries to determine which sections of the table will benefit from sorting and it automatically runs VACUUM SORT to maintain sort key order. To minimize impact to your system performance, automatic VACUUM SORT runs during periods when workloads are light.</p><p>COPY command automatically sorts and loads the data in sort key order. As a result, if you are loading an empty table using COPY command, the data is already in sort key order. If you are loading a non-empty table using COPY command, you can optimize the loads by loading data in incremental sort key order because VACUUM SORT will not be needed when your load is already in sort key order</p><p>For example, <strong>orderdate</strong> is the sort key on orders table. If you always load data into orders table where <strong>orderdate</strong> is the current date, since current date is forward incrementing, data will always be loaded in incremental sortkey(orderdate) order. Hence, in that case VACUUM SORT will not be needed for orders table.</p><p>If you need to run VACUUM SORT, you can still manually run it as shown below. Copy the following command and run it.</p><pre>vacuum sort only orders;</pre><p><em>Vacuum recluster:</em></p><p>Use VACUUM recluster whenever possible for manual VACUUM operations. This is especially important for large objects with frequent ingestion and queries that access only the most recent data. Vacuum recluster only sorts the portions of the table that are unsorted and hence runs faster. Portions of the table that are already sorted are left intact. This command doesn’t merge the newly sorted data with the sorted region. It also doesn’t reclaim all space that is marked for deletion. In order to run vacuum recluster on orders, copy the following command and run it.</p><pre>vacuum recluster orders;</pre><p><em>Vacuum boost:</em></p><p>Boost runs the VACUUM command with additional compute resources, as they’re available. With the BOOST option, VACUUM operates in one window and blocks concurrent deletes and updates for the duration of the VACUUM operation. Note that running vacuum with the BOOST option contends for system resources, which might affect performance of other queries. As a result, it is recommended to run the VACUUM BOOST when the load on the system is light, such as during maintenance operations. In order to run vacuum recluster on orders table in boost mode, copy the following command and run it.</p><pre>vacuum recluster orders boost;</pre><p>With ETL, data transformation happens in a middle-tier ETL server before loading it on the Data Warehouse. The following diagram illustrates the ETL workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/0*ZOYB4AAZvfezEUsl" /></figure><p>With ELT, data transformation happens in the target data warehouse rather than requiring a middle-tier ETL server. This approach takes advantage of Redshift database engines that support massively parallel processing (MPP). The following diagram illustrates the ELT workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5WPEhYOnlvnjX59y" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9e0550002cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Protection plans in Amazon GuardDuty]]></title>
            <link>https://medium.com/@tdhtp2016/protection-plans-in-amazon-guardduty-0631608c2342?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/0631608c2342</guid>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Thu, 17 Apr 2025 10:16:27 GMT</pubDate>
            <atom:updated>2025-04-17T10:16:27.147Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*z3kMUXUN3tBbquW1U3LVZA.png" /></figure><p><strong>Amazon GuardDuty</strong> protection types when enabled, starts processing the corresponding foundational data sources within your AWS environment. GuardDuty uses these data sources to process a stream of events, such as VPC flow logs, DNS logs, and AWS CloudTrail event and management logs. It then analyzes these events to identify potential security threats and generates findings in your account.</p><p>In addition to log data sources, GuardDuty can use additional data from other AWS services in your AWS environment to monitor and analyze for potential security threats.</p><p>Amazon GuardDuty uses the foundational data sources to detect communication with known malicious domains and IP addresses and identify anomalous behavior. While in transit from these sources to GuardDuty, all of the log data is encrypted. GuardDuty extracts various fields from these logs sources for profiling and anomaly detection, and then discards these logs.</p><p>Currently Amazon GuardDuty, provides 5 different protection plans:</p><ul><li>GuardDuty EKS Protection.</li><li>GuardDuty Lambda Protection.</li><li>GuardDuty Malware Protection.</li><li>GuardDuty RDS Protection.</li><li>GuardDuty S3 Protection.</li></ul><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/newlabs/enableingguarddutyprotection/guardduty-overview#s3-protection.">S3 Protection</a></h3><ol><li>Navigate to the <a href="https://us-east-1.console.aws.amazon.com/guardduty/home?region=us-east-1">Amazon GuardDuty </a>service.</li><li>Choose <strong>S3 Protection</strong>. S3 protections gets enable by default when you enable the service and provides a 30 day free trail.</li></ol><p>S3 protection enables Amazon GuardDuty to monitor object-level API operations to identify potential security risks for data within your S3 buckets.</p><p>GuardDuty monitors threats against your Amazon S3 resources by analyzing AWS CloudTrail management events and CloudTrail S3 data events. These data sources monitor different kinds of activity, for example, CloudTrail management events for S3 include operations that list or configure S3 buckets, such as ListBuckets, DeleteBuckets, and PutBucketReplication. Examples of data events for S3 include object-level API operations, such as GetObject, ListObjects, DeleteObject, and PutObject.</p><p>GuardDuty monitoring of CloudTrail management events is on by default for all accounts that have enabled GuardDuty and is not configurable. CloudTrail S3 data event logs are a configurable data source in GuardDuty. By default, S3 protection is enabled for new detectors, for accounts created before the addition of S3 protection this data source must be enabled manually. The processes for enabling or disabling S3 data event monitoring is covered in this topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z2HC_hWQqPfIExuS" /></figure><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/newlabs/enableingguarddutyprotection/guardduty-overview#eks-protection">EKS Protection</a></h3><ol><li>Choose <strong>EKS Protection</strong>. EKS protections gets enable by default when you enable the service and provides a 30 day free trail.</li></ol><p>EKS Protection in Amazon GuardDuty provides threat detection coverage to help you protect Amazon EKS clusters within your AWS environment. EKS Protection includes EKS Audit Log Monitoring and EKS Runtime Monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hASeX69MOx6vRi3w" /></figure><p><strong>EKS Audit Log Monitoring</strong> EKS Audit Log Monitoring helps you detect potentially suspicious activities in EKS clusters within Amazon Elastic Kubernetes Service (Amazon EKS). EKS Audit Log Monitoring uses Kubernetes audit logs to capture chronological activities from users, applications using the Kubernetes API, and the control plane.</p><p><strong>EKS Runtime Monitoring</strong> EKS Runtime Monitoring uses operating system-level events to help you detect potential threats in Amazon EKS nodes and containers within your Amazon EKS clusters.</p><p>2. EKS Runtime Monitoring requires a new GuardDuty security agent (EKS add-on) that adds runtime visibility into individual EKS workloads, for example, file access, process execution, and network connections.</p><p>3. You can choose to either deploy the agent automatically (managed by the GuardDuty service on you behalf) or manually; in this case you will need to coordinate the deployment of the GuardDuty security agent software within your EKS clusters, across all accounts and AWS Regions where this feature is available. This will also require you to support secure data flow while monitoring for and addressing coverage gaps as new clusters and workloads are deployed.</p><p>4. If you want to modify this, you can do so on the <strong>Edit</strong> option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lPT_E79PKx32JiYe" /></figure><p>5. Here you can enable/disable protections and choose the type of management for the required agent for runtime monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7mC0QkZuCGVZk2Jx" /></figure><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/newlabs/enableingguarddutyprotection/guardduty-overview#malware-protection">Malware Protection</a></h3><p>Malware Protection helps you detect the potential presence of malware by scanning the Amazon Elastic Block Store (Amazon EBS) volumes that are attached to the Amazon Elastic Compute Cloud (Amazon EC2) instances and container workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qjdi70A4U-KL7uNM" /></figure><p>Malware Protection offers two types of scans to detect potentially malicious activity in your Amazon EC2 instances and container workloads, <strong>GuardDuty-initiated</strong> malware scan and <strong>On-demand malware scan</strong>.</p><p><strong>GuardDuty-initiated</strong> are triggered whenever GuardDuty detects malicious activity that indicates the potential presence of malware in your Amazon EC2 instance or container. This is not enabled by default when you enable Amazon GuardDuty.</p><ol><li>To enable, click on <strong>Enable</strong> and <strong>Confirm</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*re9CtTMOY_ZPNiU8" /></figure><p>2. You will see the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D9VeS3DiXp9bV-Nz" /></figure><p>For <strong>On-demand malware scan</strong>, Malware Protection provides scan options where you can decide if you want to include or exclude specific Amazon EC2 instances and container workloads at the time of scanning. It also provides an option to retain the snapshots of Amazon EBS volumes attached to the Amazon EC2 instances or container workloads, in your GuardDuty accounts. The snapshots get retained only when malware is found and Malware Protection findings are generated.</p><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/newlabs/enableingguarddutyprotection/guardduty-overview#rds-protection">RDS Protection</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7yUQrNBDStZn_Lgj" /></figure><p>RDS Protection in Amazon GuardDuty analyzes and profiles RDS login activity for potential access threats to your Amazon Aurora databases (Amazon Aurora MySQL-Compatible Edition and Aurora PostgreSQL-Compatible Edition). This feature allows you to identify potentially suspicious login behavior. RDS Protection doesn’t require additional infrastructure; it is designed so as not to affect the performance of your database instances.</p><p>When RDS Protection detects a potentially suspicious or anomalous login attempt that indicates a threat to your database, GuardDuty generates a new finding with details about the potentially compromised database.</p><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/newlabs/enableingguarddutyprotection/guardduty-overview#lambda-protection">Lambda Protection</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gt-BdKjhsP3v41VC" /></figure><p>Lambda Protection helps you identify potential security threats when an AWS Lambda function gets invoked in your AWS environment. When you enable Lambda Protection, GuardDuty starts monitoring Lambda network activity logs, starting with VPC Flow Logs from all Lambda functions for account, including those logs that don’t use VPC networking, and are generated when the Lambda function gets invoked. If GuardDuty identifies suspicious network traffic that is indicative of the presence of a potentially malicious piece of code in your Lambda function, GuardDuty will generate a finding.</p><h3><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/newlabs/enableingguarddutyprotection/guardduty-overview#summary-view">Summary View</a></h3><p>1. As we just rencently enabled Amazon GuardDuty, we will probably not have any findings. So let’s proceed to generate sample findings.</p><p>2. Choose <strong>Settings</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jHn2j8KkgN8S16FK" /></figure><p>3. Click on <strong>Generate sample findings</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Le1Vm96OlHOy-ews" /></figure><p>4. Now navigate to <strong>Summary</strong>, here you will have a dashboard of insights of the findings we just generated.</p><p>The Summary dashboard provides an aggregated view of the GuardDuty findings generated in your AWS account in the current Region. Presently, the dashboard supports a volume of up to 5,000 findings.</p><p>Out of a volume of up to last 5,000 GuardDuty findings, the summary dashboard with Accounts with most findings, Resources with most findings, and Least occurring findings shows the data based off of the top 5 results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0631608c2342" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enabling Amazon GuardDuty via AWS Systems Manager]]></title>
            <link>https://medium.com/@tdhtp2016/enabling-amazon-guardduty-via-aws-systems-manager-0c6124ecb749?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c6124ecb749</guid>
            <category><![CDATA[system-manager]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-guardduty]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Thu, 17 Apr 2025 10:07:32 GMT</pubDate>
            <atom:updated>2025-04-17T10:07:32.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*ZN_tuSDlaPQvuvPQt05Xqw.png" /></figure><p><a href="https://aws.amazon.com/guardduty/">Amazon GuardDuty </a>is a threat detection service that continuously monitors your AWS accounts and workloads for malicious activity and delivers detailed security findings for visibility and remediation.</p><p><a href="https://aws.amazon.com/systems-manager/">AWS Systems Manager </a>is a secure end-to-end management solution for resources on AWS and in multicloud and hybrid environments.</p><h4>Deploying an AWS System Manager document to enable Amazon GuardDuty</h4><p><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/documents.html">AWS Systems Manager Documents </a>, defines the actions that Systems Manager performs on your managed instances. Systems Manager includes more than 100 pre-configured documents that you can use by specifying parameters at runtime. You can find pre-configured documents in the Systems Manager Documents console by choosing the Owned by Amazon tab, or by specifying Amazon for the Owner filter when calling the ListDocuments API operation. Documents use JavaScript Object Notation (JSON) or YAML, and they include steps and parameters that you specify.</p><h4><a href="https://catalog.us-east-1.prod.workshops.aws/event/dashboard/en-US/workshop/newlabs/enableingguarddutyprotection/deploy-ssm-doc#create-an-aws-systems-manager-role-to-enable-amazon-guardduty">Create an AWS Systems Manager role to enable Amazon GuardDuty</a></h4><ol><li>Navigate to the <a href="https://console.aws.amazon.com/config/home">AWS Config console</a></li><li>Navigate to the <a href="https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles">IAM-Roles </a>service.</li><li>Choose <strong>Create Role</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*AJByjYPgOUVm4VYf" /></figure><p>4. Choose Trusted entity type <strong>AWS Service</strong> and in the dropdown select <strong>Systems Manager</strong> as shown below and click <strong>Next</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/0*XzLCM1fhb_vpzhxg" /></figure><p>5. Choose <strong>Create Policy</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/0*DP6-AWPybRxTLq31" /></figure><p>6. In the Permissions step, select <strong>JSON</strong> and copy paste to following policy in the editor as follows and click <strong>Next</strong>:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;guardduty:GetFindings&quot;,<br>                &quot;guardduty:ListThreatIntelSets&quot;,<br>                &quot;guardduty:GetThreatIntelSet&quot;,<br>                &quot;guardduty:CreateDetector&quot;,<br>                &quot;guardduty:ListFindings&quot;,<br>                &quot;guardduty:DescribeMalwareScans&quot;,<br>                &quot;guardduty:GetAdministratorAccount&quot;,<br>                &quot;guardduty:GetDetector&quot;,<br>                &quot;guardduty:GetRemainingFreeTrialDays&quot;,<br>                &quot;guardduty:ListInvitations&quot;,<br>                &quot;guardduty:GetMemberDetectors&quot;,<br>                &quot;guardduty:DescribePublishingDestination&quot;,<br>                &quot;guardduty:ListPublishingDestinations&quot;,<br>                &quot;guardduty:GetFilter&quot;,<br>                &quot;guardduty:ListTagsForResource&quot;,<br>                &quot;guardduty:ListMembers&quot;,<br>                &quot;guardduty:ListIPSets&quot;,<br>                &quot;guardduty:ListCoverage&quot;,<br>                &quot;guardduty:ListDetectors&quot;,<br>                &quot;guardduty:GetMasterAccount&quot;,<br>                &quot;guardduty:GetCoverageStatistics&quot;,<br>                &quot;guardduty:GetIPSet&quot;,<br>                &quot;guardduty:GetMembers&quot;,<br>                &quot;guardduty:GetMalwareScanSettings&quot;,<br>                &quot;guardduty:GetUsageStatistics&quot;,<br>                &quot;guardduty:ListOrganizationAdminAccounts&quot;,<br>                &quot;guardduty:DescribeOrganizationConfiguration&quot;,<br>                &quot;guardduty:ListFilters&quot;,<br>                &quot;guardduty:GetFindingsStatistics&quot;,<br>                &quot;guardduty:GetInvitationsCount&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pl0xUTuCpL-wq7IZ" /></figure><p>7. Give the policy a proper name and click <strong>Create policy</strong>.</p><p>8. Then back in the Permissions step of the role creation, choose the policy recently created and click <strong>Next</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UiJtlm2is8LrAf5k" /></figure><p>9. Give the role a proper name and click <strong>Create Role</strong>.</p><p>10. Navigate to the <a href="https://us-east-1.console.aws.amazon.com/systems-manager/home?region=us-east-1">AWS Systems Manager </a>service.</p><p>11. In the left navigation pane, you will find all AWS Systems Manager features that allows us to centralize operation activities across our AWS environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H8VSKFiuEmJqvCBT" /></figure><p>12. For this particular activity, we are going to leverage <strong>Documents</strong> under <strong>Shared Resources</strong> in the bottom of the left navigation pane. Documents</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/0*AXtw2c-PLQBf-PZB" /></figure><p>13. In the tab for <strong>Owned by Amazon</strong> documents, type in the Search bar <strong>guardduty</strong> and click in the document to proceed as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/0*uU7IMACMKyWj5sIl" /></figure><p>14. This document will require one Input Parameter (the role we created before) and will generate an Output parameter, that is the Amazon GuardDuty detector for the region we are in. So, after reviewing the steps; click on <strong>Execute automation</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3ApxL5Q7Usv1CdZQ" /></figure><p>15. Choose <strong>Simple execution</strong>, the role you created in earlier steps from the dropdown menu in the <strong>Input parameters</strong> section and click on <strong>Execute</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xYrNrwVExZL_S5bN" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*COSqlDxe3ObgVfIc" /></figure><p>16. After execution, you should see a <strong>Success</strong> status overall and in the API calls executed. In case it fails, the console will provide you an error code and message. Most of the errors here could be lack of permissions in the role or a detector already enabled in Amazon GuardDuty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EHKRlQoSdWGQ5Coh" /></figure><p>We have deployed AWS System Manager documents and enable Amazon GuardDuty leveraging this automation feature.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c6124ecb749" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Disaster Recovery]]></title>
            <link>https://medium.com/@tdhtp2016/aws-disaster-recovery-842b34b34864?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/842b34b34864</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[disaster-recovery]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Wed, 06 Nov 2024 09:26:09 GMT</pubDate>
            <atom:updated>2024-11-06T09:26:09.016Z</atom:updated>
            <content:encoded><![CDATA[<h3>Backup and Restore</h3><p>In this module, you will go through the Backup and Restore disaster recovery strategy. To learn more about this disaster recovery strategy, you can review this <a href="https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-ii-backup-and-restore-with-rapid-recovery/">Disaster Recovery blog </a>.</p><p>Backup and Restore disaster recovery strategy has <a href="https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/disaster-recovery-dr-objectives.html">Recovery Point Objective(RPO) / Recovery Time Objective (RTO) </a>within <em>hours</em>. For the backup and restore strategy secondary region, all data, infrastructure and services need to be provisioned.</p><p>Your application is currently deployed in the primary region <strong>N. Virginia (us-east-1)</strong>. <strong>Oregon (us-west-2)</strong> will be your secondary</p><p>Your test application is a Spring Boot Java application deployed on a single <a href="https://aws.amazon.com/ec2">Amazon Elastic Compute Cloud (EC2) </a>instance using a public subnet. Your datastore is an <a href="https://aws.amazon.com/rds/">Amazon RDS </a>MySQL database with a frontend written using bootstrap and hosted in <a href="https://aws.amazon.com/pm/serv-s3">Amazon Simple Storage Service (S3) </a>.</p><p>This module takes advantage of <a href="https://aws.amazon.com/backup/">AWS Backup </a>which will be your single pane of glass to backup, copy and restore your Amazon EC2 instance and Amazon RDS database to the secondary region.</p><p>You will use <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html#crr-scenario">Amazon S3 Cross-Region Replication (CRR) </a>to replicate your S3 objects to the secondary region.</p><p><a href="https://aws.amazon.com/cloudformation/">CloudFormation </a>will be used to configure the infrastructure and deploy the application. Provisioning your infrastructure with infrastructure as code (IaC) methodologies is a best practice. CloudFormation is an easy way to speed up cloud provisioning with infrastructure as code.</p><p>Prior experience with the AWS Console and Linux command line is helpful but not required.</p><p>Because this workload has only <strong>one</strong> EC2 instance that is deployed in only <strong>one</strong> Availability Zone, this architecture does not meet the AWS Well Architected Framework best practices for running highly available production applications but suffices for this workshop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/0*CMLq48dwI_qaxCHz" /></figure><h3>Pilot Light</h3><p>In this module, you will go through the Pilot Light disaster recovery strategy. To learn more about this disaster recovery strategy, you can review this <a href="https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iii-pilot-light-and-warm-standby/">Disaster Recovery blog </a>.</p><p>Pilot Light disaster recovery strategy has <a href="https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/disaster-recovery-dr-objectives.html">Recovery Point Objective(RPO) / Recovery Time Objective (RTO) </a><em>within tens of minutes</em>. For the pilot light strategy secondary region, the data is live and core infrastructure is provisioned, but the services are either idle or absent.</p><p>Your application is currently deployed in the primary region <strong>N. Virginia (us-east-1)</strong>. <strong>N. California (us-west-1)</strong> will be your secondary</p><p>Your test application is a Spring Boot Java application deployed on a single <a href="https://aws.amazon.com/ec2">Amazon Elastic Compute Cloud (EC2) </a>instance using a public subnet. Your datastore is an <a href="https://aws.amazon.com/rds/aurora/">Amazon Aurora </a>MySQL database with a frontend written using bootstrap and hosted in <a href="https://aws.amazon.com/pm/serv-s3">Amazon Simple Storage Service (S3) </a>.</p><p>This module takes advantage of <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">Amazon Machine Images (AMI) </a>which you will use to launch your Amazon EC2 instance and <a href="https://aws.amazon.com/rds/aurora/global-database/">Amazon Aurora Global Database </a>to replicate your Amazon Aurora MySQL data to your secondary region.</p><p><a href="https://aws.amazon.com/cloudformation/">CloudFormation </a>will be used to configure the infrastructure and deploy the application. Provisioning your infrastructure with infrastructure as code (IaC) methodologies is a best practice. CloudFormation is an easy way to speed up cloud provisioning with infrastructure as code.</p><p>Prior experience with the AWS Console and Linux command line are helpful but not required.</p><p>Because this workload has only <strong>one</strong> EC2 instance that is deployed in only <strong>one</strong> Availability Zone, this architecture does not meet the AWS Well Architected Framework best practices for running highly available production applications but suffices for this workshop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/0*iG1Bf27esW7773hU" /></figure><h3>Warm Standby</h3><p>In this module, you will go through the Warm Standby disaster recovery strategy. To learn more about this disaster recovery strategy, you can review this <a href="https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iii-pilot-light-and-warm-standby/">Disaster Recovery blog </a>.</p><p>Warm Standby disaster recovery strategy has <a href="https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/disaster-recovery-dr-objectives.html">Recovery Point Objective(RPO) / Recovery Time Objective (RTO) </a><em>within minutes</em>. For the warm standby strategy secondary region, the data is live and core infrastructure is provisioned and the services are running at a reduced capacity.</p><p>Our application is currently deployed in our primary region <strong>N. Virginia (us-east-1)</strong> and we will use <strong>Oregon (us-west-2)</strong> as our secondary region.</p><p>Your test application is a Spring Boot Java application deployed using <a href="https://aws.amazon.com/elasticloadbalancing/">Elastic Load Balancing </a>and two <a href="https://aws.amazon.com/ec2">Amazon Elastic Compute Cloud (EC2) </a>instances using a public subnet. Datastore is an <a href="https://aws.amazon.com/rds/aurora/">Amazon Aurora </a>MySQL database with a frontend written using bootstrap and hosted in <a href="https://aws.amazon.com/pm/serv-s3">Amazon Simple Storage Service (S3) </a>.</p><p>This module takes advantage of <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html">Auto Scaling Groups (ASG) </a>which will allow us to scale out in the secondary region compute instances during failover. <a href="https://aws.amazon.com/rds/aurora/global-database/">Amazon Aurora Global Database </a>is used to replicate your Amazon Aurora MySQL data to the secondary region. You are also taking advantage of Amazon Aurora read replica <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-write-forwarding.html">write forwarding </a>. With this feature enabled, writes can be sent to a read replica in a secondary region, and will be seamlessly forwarded to the writer in the primary region over a secure communication channel.</p><p><a href="https://aws.amazon.com/cloudformation/">CloudFormation </a>will be used to configure the infrastructure and deploy the application. Provisioning your infrastructure with infrastructure as code (IaC) methodologies is a best practice. CloudFormation is an easy way to speed up cloud provisioning with infrastructure as code.</p><p>Prior experience with the AWS Console and Linux command line are helpful but not required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/0*O-R38FvupRS28n0F" /></figure><h3>Hot Standby</h3><p>In this module, you will go through the Hot Standby disaster recovery strategy. To learn more about this disaster recovery strategy, you can review this <a href="https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iii-pilot-light-and-warm-standby/">Disaster Recovery blog </a>.</p><p>Hot Standby disaster recovery strategy has <a href="https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/disaster-recovery-dr-objectives.html">Recovery Point Objective(RPO) / Recovery Time Objective (RTO) </a><em>in almost real time</em>. For the hot standby strategy secondary region, the data is live, core infrastructure is provisioned and the services are running at full production capacity.</p><p>Your application is currently deployed in the primary region <strong>N. Virginia (us-east-1)</strong>. <strong>Oregon (us-west-2)</strong> will be your secondary region.</p><p>Your test application is a Spring Boot Java application deployed on a single <a href="https://aws.amazon.com/ec2">Amazon Elastic Compute Cloud (EC2) </a>instance using a public subnet. Your datastore is an <a href="https://aws.amazon.com/rds/aurora/">Amazon Aurora </a>MySQL database which has user data. Your test application is also deployed using <a href="https://aws.amazon.com/api-gateway/">Amazon API Gateway </a>and <a href="https://aws.amazon.com/lambda/">AWS Lambda </a>. Your datastore is <a href="https://aws.amazon.com/dynamodb">Amazon DynamoDB </a>which has shopping cart data. The frontend is written using bootstrap and hosted in <a href="https://aws.amazon.com/pm/serv-s3">Amazon Simple Storage Service (S3) </a>.</p><p>Your test application is using two datastores, Amazon Aurora and Amazon DynamoDB to showcase the Disaster Recovery features of each. You are using Amazon DynamoDB to store the state of the shopping cart and you are using Amazon Aurora to store the unicorn product catalog. When you initiate failover, for the Amazon Aurora database you will not need to promote the read replica in the recovery region since you are only reading the product catalog. This can decrease recovery time since no database promotion is necessary. In the failover case of Amazon DynamoDB you have configured Global Tables which allows for multiple writers so no action is needed. For your workloads, you would choose the right datastore for your use case.</p><p>This module takes advantage of <a href="https://aws.amazon.com/cloudfront/">Amazon CloudFront </a>which you will use as your content delivery network. You are also taking advantage of <a href="https://aws.amazon.com/rds/aurora/global-database/">Amazon Aurora Global Database </a>to replicate your Amazon Aurora MySQL data to your secondary region and <a href="https://aws.amazon.com/dynamodb/global-tables/">Amazon DynamoDB Global Tables </a>to replicate your DynamoDB data to your secondary region.</p><p><a href="https://aws.amazon.com/cloudformation/">CloudFormation </a>will be used to configure the infrastructure and deploy the application. Provisioning your infrastructure with infrastructure as code (IaC) methodologies is a best practice. CloudFormation is an easy way to speed up cloud provisioning with infrastructure as code.</p><p>Prior experience with the AWS Console and Linux command line are helpful but not required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/0*lvifPuyYkwNt6s-6" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=842b34b34864" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform — tf.state file Deep Dive]]></title>
            <link>https://medium.com/@tdhtp2016/terraform-tf-state-file-deep-dive-6d9aa639e729?source=rss-ed98d4be1205------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d9aa639e729</guid>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Duy Hưng]]></dc:creator>
            <pubDate>Sat, 10 Aug 2024 07:29:04 GMT</pubDate>
            <atom:updated>2024-08-10T07:29:04.292Z</atom:updated>
            <content:encoded><![CDATA[<h3>Terraform — tf.state file Deep Dive</h3><h3>What Is Terraform State?</h3><p>Every time you run Terraform, it records information about what infrastructure it created in a Terraform state file.</p><p>By default, when you run Terraform in the folder /foo/bar, Terraform creates the file /foo/bar/terraform.tfstate.</p><p>This file contains a custom JSON format that records a mapping from the Terraform resources in your configuration files to the representation of those resources in the real world.</p><pre>resource &quot;aws_instance&quot; &quot;example&quot; {<br>  ami           = &quot;ami-0fb653ca2d3203ac1&quot;<br>  instance_type = &quot;t2.micro&quot;<br>}</pre><p>After running terraform apply, here is a small snippet of the contents of the<br><strong>terraform.tfstate </strong>file (truncated for readability):</p><pre>{<br>  &quot;version&quot;: 4,<br>  &quot;terraform_version&quot;: &quot;1.2.3&quot;,<br>  &quot;serial&quot;: 1,<br>  &quot;lineage&quot;: &quot;86545604-7463-4aa5-e9e8-a2a221de98d2&quot;,<br>  &quot;outputs&quot;: {},<br>  &quot;resources&quot;:<br>    [<br>      {<br>        &quot;mode&quot;: &quot;managed&quot;,<br>        &quot;type&quot;: &quot;aws_instance&quot;,<br>        &quot;name&quot;: &quot;example&quot;,<br>        &quot;provider&quot;: &#39;provider[&quot;registry.terraform.io/hashicorp/aws&quot;]&#39;,<br>        &quot;instances&quot;:<br>          [<br>            {<br>              &quot;schema_version&quot;: 1,<br>              &quot;attributes&quot;:<br>                {<br>                  &quot;ami&quot;: &quot;ami-0fb653ca2d3203ac1&quot;,<br>                  &quot;availability_zone&quot;: &quot;us-east-2b&quot;,<br>                  &quot;id&quot;: &quot;i-0bc4bbe5b84387543&quot;,<br>                  &quot;instance_state&quot;: &quot;running&quot;,<br>                  &quot;instance_type&quot;: &quot;t2.micro&quot;,<br>                  &quot;(...)&quot;: &quot;(truncated)&quot;,<br>                },<br>            },<br>          ],<br>      },<br>    ],<br>}</pre><p>Every time you run Terraform, it can fetch the latest status of this EC2 Instance from AWS and compare that to what’s in your Terraform configurations to determine what changes need to be applied.</p><p>In other words, the output of the <strong>plan </strong>command is a diff between the code on your computer and the infrastructure deployed in the real world, as discovered via IDs in the state file.</p><blockquote><strong>The State File Is a Private API</strong></blockquote><blockquote>The state file format is a private API that is meant only for internal<br>use within Terraform. You should never edit the Terraform state<br>files by hand or write code that reads them directly.</blockquote><blockquote>If for some reason you need to manipulate the state file — which<br>should be a relatively rare occurrence — use the terraform <strong>import</strong><br>or terraform <strong>state </strong>commands.</blockquote><h3>Shared Storage for State Files</h3><p>The most common technique for allowing multiple team members to access a common set of files is to put them in version control (e.g., Git). Although you should definitely store your Terraform code in version control, storing Terraform state in version control is a bad idea for the following reasons:</p><ul><li><strong>Manual error </strong>It’s too easy to forget to pull down the latest changes from version control before running Terraform or to push your latest changes to version control after running Terraform. It’s just a matter of time before someone on your team runs Terraform with out-of-date state files and as a result, accidentally rolls back or duplicates previous deployments.</li><li><strong>Locking </strong>Most version control systems do not provide any form of locking that would prevent two team members from running terraform apply on the same state file at the same time.</li><li><strong>Secrets </strong>All data in Terraform state files is stored in plain text. This is a problem because certain Terraform resources need to store sensitive data. For example, if you use the aws_db_instance resource to create a database, Terraform will store the username and password for the database in a state file in plain text, and you shouldn’t store plain text secrets in version control.</li></ul><p>Instead of using version control, the best way to manage shared storage for state files is to use Terraform’s built-in support for <strong>remote backends. </strong>A Terraform backend determines how Terraform loads and stores state. The default backend, which you’ve been using this entire time, is the local backend, which stores the state file on your local disk.</p><p><strong>Remote backends </strong>allow you to store the state file in a remote, shared store.<br>A number of remote backends are supported, including Amazon S3, Azure Storage, Google Cloud Storage, and HashiCorp’s Terraform Cloud and Terraform Enterprise.</p><p>Remote backends solve the three issues just listed:</p><ul><li><strong>Manual error </strong>After you configure a remote backend, Terraform will automatically load the state file from that backend every time you run plan or apply and it’ll automatically store the state file in that backend after each apply, so there’s no chance of manual error.</li><li><strong>Locking </strong>Most of the remote backends natively support locking. To run terraform apply, Terraform will automatically acquire a lock; if someone else is already running apply, they will already have the lock, and you will have to wait. You can run apply with the -lock-timeout=&lt;TIME&gt; parameter to instruct Terraform to wait up to TIME for a lock to be released (e.g., -lock-timeout=10m will wait for 10 minutes).</li><li><strong>Secrets </strong>Most of the remote backends natively support encryption in transit and encryption at rest of the state file. Moreover, those backends usually expose ways to configure access permissions (e.g., using IAM policies with an Amazon S3 bucket), so you can control who has access to your state files and the secrets they might contain. It would be better still if Terraform natively supported encrypting secrets within the state file, but these remote backends reduce most of the security concerns, given that at least the state file isn’t stored in plain text on disk anywhere.</li></ul><p>If you’re using Terraform with AWS, Amazon S3 (Simple Storage Service), which is Amazon’s managed file store, is typically your best bet as a remote backend for the following reasons:</p><ul><li>It’s a managed service, so you don’t need to deploy and manage extra infrastructure to use it.</li><li>It’s designed for 99.999999999% durability and 99.99% availability, which means you don’t need to worry too much about data loss or outages.</li><li>It supports encryption, which reduces worries about storing sensitive data in state files. You still have to be very careful who on your team can access the S3 bucket, but at least the data will be encrypted at rest (Amazon S3 supports server-side encryption using AES-256) and in transit (Terraform uses TLS when talking to Amazon S3).</li><li>It supports locking via DynamoDB.</li><li>It supports versioning, so every revision of your state file is stored, and you can roll back to an older version if something goes wrong.</li><li>It’s inexpensive, with most Terraform usage easily fitting into the free tier.</li></ul><pre>terraform {<br>  backend &quot;s3&quot; {<br>    # Replace this with your bucket name!<br>    bucket = &quot;terraform-up-and-running-state&quot;<br>    key    = &quot;global/s3/terraform.tfstate&quot;<br>    region = &quot;us-east-2&quot;<br>    # Replace this with your DynamoDB table name!<br>    dynamodb_table = &quot;terraform-up-and-running-locks&quot;<br>    encrypt        = true<br>  }<br>}</pre><h4>Limitations with Terraform’s Backends</h4><p>Terraform’s backends have a few limitations and gotchas that you need to be aware of. The first limitation is the chicken-and-egg situation of using Terraform to create the S3 bucket where you want to store your Terraform state. To make this work, you had to use a two-step process:</p><ol><li>Write Terraform code to create the S3 bucket and DynamoDB table and deploy that code with a local backend.</li><li>Go back to the Terraform code, add a remote backend configuration to it to use the newly created S3 bucket and DynamoDB table, and run terraform init to copy your local state to S3.</li></ol><p>If you ever wanted to delete the S3 bucket and DynamoDB table, you’d have to do this two-step process in reverse:</p><ol><li>Go to the Terraform code, remove the backend configuration, and rerun <strong>terraform init </strong>to copy the Terraform state back to your local disk.</li><li>Run <strong>terraform destroy </strong>to delete the S3 bucket and DynamoDB table.</li></ol><p>This two-step process is a bit awkward, but the good news is that you can share a single S3 bucket and DynamoDB table across all of your Terraform code, so you’ll probably only need to do it once (or once per AWS account if you have multiple accounts). After the S3 bucket exists, in the rest of your Terraform code, you can specify the backend configuration right from the start without any extra steps.</p><p>The second limitation is more painful: the backend block in Terraform does not allow you to use any variables or references. The following code will not work:</p><pre>terraform {<br>  backend &quot;s3&quot; {<br>    bucket         = var.bucket<br>    region         = var.region<br>    dynamodb_table = var.dynamodb_table<br>    key            = &quot;example/terraform.tfstate&quot;<br>    encrypt        = true<br>  }<br>}</pre><p>This means that you need to manually copy and paste the S3 bucket name, region, DynamoDB table name, etc. into every one of your Terraform modules.</p><p>Even worse, you must very carefully not copy and paste the key value, but ensure a unique key for every Terraform module you deploy so that you don’t accidentally overwrite the state of some other module! Having to do lots of copy-and-pastes and lots of manual changes is error prone, especially if you need to deploy and manage many Terraform modules<br>across many environments.</p><p>One option for reducing copy-and-paste is to use partial configurations, where you omit certain parameters from the backend configuration in your Terraform code and instead pass those in via -backend-config CLI arguments when calling terraform init. For example, you could extract the repeated backend arguments, such as bucket and region, into a separate file called backend.hcl:</p><pre># backend.hcl<br>bucket = &quot;terraform-up-and-running-state&quot;<br>region = &quot;us-east-2&quot;<br>dynamodb_table = &quot;terraform-up-and-running-locks&quot;<br>encrypt = true</pre><p>Only the key parameter remains in the Terraform code, since you still need to set a different key value for each module:</p><pre># Partial configuration. The other settings (e.g., bucket, region) will be<br># passed in from a file via -backend-config arguments to &#39;terraform init&#39;<br>terraform {<br>  backend &quot;s3&quot; {<br>    key = &quot;example/terraform.tfstate&quot;<br>  }<br>}</pre><p>To put all your partial configurations together, run terraform init with the<br>-backend-config argument:</p><pre>$ terraform init -backend-config=backend.hcl</pre><p>Terraform merges the partial configuration in backend.hcl with the partial configuration in your Terraform code to produce the full configuration used by your module.</p><p>You can use the same <strong>backend.hcl </strong>file with all of your modules, which reduces duplication considerably; however, you’ll still need to manually set a unique key value in every module.</p><p>Another option for reducing copy-and-paste is to use <a href="https://github.com/gruntwork-io/terragrunt">Terragrunt</a>, an open source tool that tries to fill in a few gaps in Terraform. Terragrunt can help you keep your entire backend configuration <strong>DRY (Don’t Repeat Yourself) </strong>by defining all the basic backend settings (bucket name, region, DynamoDB table name) in one file and automatically setting the key argument to the relative folder path of the module.</p><h3>Isolating State Files</h3><p>With a remote backend and locking, collaboration is no longer a problem. However, there is still one more problem remaining: <strong>isolation</strong>.</p><p>When you first start using Terraform, you might be tempted to define all of your infrastructure in a single Terraform file or a single set of Terraform files in one folder. The problem with this approach is that all of your Terraform state is now stored in a single file, too, and a mistake anywhere could break everything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*yPnD5nExqgW2ZaxIh4Rs9Q.png" /></figure><p>The whole point of having separate environments is that they are isolated from one another, so if you are managing all the environments from a single set of Terraform configurations, you are breaking that isolation. Just as a ship has bulkheads that act as barriers to prevent a leak in one part of the ship from immediately flooding all the others, you should have “bulkheads” built into your Terraform design.</p><h4>Isolation via Workspaces</h4><p>Terraform workspaces allow you to store your Terraform state in multiple, separate, named workspaces. Terraform starts with a single workspace called “default,” and if you never explicitly specify a workspace, the default workspace is the one you’ll use the entire time. To create a new workspace or switch between workspaces, you use the <strong>terraform workspace </strong>commands.</p><h4>Isolation via File Layout</h4><p>To achieve full isolation between environments, you need to do the following:</p><ul><li>Put the Terraform configuration files for each environment into a separate folder. For example, all of the configurations for the staging environment can be in a folder called stage and all the configurations for the production environment can be in a folder called prod.</li><li>Configure a different backend for each environment, using different authentication mechanisms and access controls: e.g., each environment could live in a separate AWS account with a separate S3 bucket as a backend.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*M2H3PvWzpbQ8Jao4KArrEw.png" /></figure><p>With this approach, the use of separate folders makes it much clearer which environments you’re deploying to, and the use of separate state files, with separate authentication mechanisms, makes it significantly less likely that a screw up in one environment can have any impact on another.<br>In fact, you might want to take the isolation concept beyond environments and down to the “component” level, where a component is a coherent set of resources that you typically deploy together. For example, after you’ve set up the basic network topology for your infrastructure — in AWS lingo, your Virtual Private Cloud (VPC) and all the associated subnets, routing rules, VPNs, and network ACLs — you will probably change it only once every few months, at most.</p><p>On the other hand, you might deploy a new version of a web server multiple times per day. If you manage the infrastructure for both the VPC component and the web server component in the same set of Terraform configurations, you are unnecessarily putting your entire network topology at risk of breakage (e.g., from a simple typo in the code or someone accidentally running the wrong command) multiple times per day.</p><p>Therefore, I recommend using separate Terraform folders (and therefore separate state files) for each environment (staging, production, etc.) and for each component (VPC, services, databases) within that environment. To see what this looks like in practice, let’s go through the recommended file layout for Terraform projects.</p><h3>The terraform_remote_state Data Source</h3><p>You have used data sources to fetch read-only information from AWS,<br>such as the aws_subnets data source, which returns a list of subnets in your VPC. There is another data source that is particularly useful when working with state: <strong>terraform_remote_state</strong>.</p><p>You can use this data source to fetch the Terraform state file stored by another set of Terraform configurations in a completely read-only<br>manner. Let’s go through an example. Imagine that your web server cluster needs to communicate with a MySQL database. Running a database that is scalable, secure, durable, and highly available is a lot of work. Again, you can let AWS take care of it for you, this time by using Amazon’s Relational Database Service (RDS). RDS supports a variety of databases, including MySQL, PostgreSQL, SQL Server, and Oracle</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vp1TQbthRd8kuUmN6Y0rBA.png" /></figure><p>You might not want to define the MySQL database in the same set of configuration files as the web server cluster, because you’ll be deploying updates to the web server cluster far more frequently and don’t want to risk accidentally breaking the database each time you do so. Therefore, your first step should be to create a new folder at <strong>stage/data-stores/mysql </strong>and create the basic Terraform files <strong>(main.tf, variables.tf, outputs.tf)</strong> within it,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*0V9xPqnr3MYBttxkf3ukNA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d9aa639e729" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>