<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oluwatobi Akinlade on Medium]]></title>
        <description><![CDATA[Stories by Oluwatobi Akinlade on Medium]]></description>
        <link>https://medium.com/@tobi.akinlade.co?source=rss-96a31a8dcfdd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bbjCLXXN6fLwy8--sfCcgQ.jpeg</url>
            <title>Stories by Oluwatobi Akinlade on Medium</title>
            <link>https://medium.com/@tobi.akinlade.co?source=rss-96a31a8dcfdd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:48:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tobi.akinlade.co/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part 1: Infrastructure, Kubernetes Deployment, Vault Secrets, Service Security & Auto-Scaling]]></title>
            <link>https://medium.com/@tobi.akinlade.co/part-1-infrastructure-kubernetes-deployment-vault-secrets-service-security-auto-scaling-18f6b39630ee?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/18f6b39630ee</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Tue, 10 Mar 2026 22:32:39 GMT</pubDate>
            <atom:updated>2026-03-10T22:32:39.224Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Production-Grade Microservice Platform on AWS EKS — A Step-by-Step Guide</h3><p><em>By Tobi Akinlade · DevOps Engineer</em></p><h3>Platform Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoxoxGojAQ_Nr1hRzc0r-Q.png" /></figure><h3>What You’ll Build</h3><p>By the end of this guide, you’ll have a fully production-grade microservice platform running on AWS. Not a tutorial that works in isolation — a platform built with the same patterns used in a large enterprise.</p><p><strong>Infrastructure (Terraform):</strong></p><ul><li>3-tier VPC (public/private/isolated) across 3 Availability Zones</li><li>EKS cluster with IRSA (IAM Roles for Service Accounts)</li><li>RDS PostgreSQL in isolated subnets</li><li>DynamoDB tables with a single-table design and 3 GSIs</li><li>ElastiCache Redis with cache-aside pattern</li><li>SNS/SQS event-driven messaging with Dead Letter Queues</li><li>ECR repositories, S3 buckets, VPC endpoints</li></ul><p><strong>Kubernetes:</strong></p><ul><li>4 microservices + API gateway deployed via Helm</li><li>HashiCorp Vault with dynamic database credentials, Transit encryption, and audit logging</li><li>VPC CNI network policies (default-deny + explicit allow)</li><li>OPA Gatekeeper admission control</li><li>Horizontal Pod Autoscaler per service</li></ul><h3>Prerequisites</h3><p>Before starting, install these tools:</p><pre># Verify you have these installed<br>terraform --version    # &gt;= 1.5<br>aws --version          # AWS CLI v2<br>kubectl version        # Latest stable<br>helm version           # &gt;= 3.12<br>docker --version       # Docker Desktop or Docker Engine</pre><p>You’ll also need:</p><ul><li>An AWS account with admin access (or sufficient IAM permissions)</li><li>AWS CLI configured: aws configure (set region to eu-west-2)</li></ul><h3>Step 1: Project Setup</h3><p>Clone or create the project structure:</p><pre>git clone https://gitlab.com/tobiakinlade/jobboard-platform.git</pre><p>The project follows a modular structure. Each Terraform module handles one infrastructure concern:</p><pre>terraform/<br>├── main.tf              ← Wires all modules together<br>├── variables.tf         ← Input variables (what changes per environment)<br>├── outputs.tf           ← Values other tools need (endpoints, ARNs)<br>├── providers.tf         ← AWS, Kubernetes, Helm provider config<br>├── environments/<br>│   └── dev/<br>│       ├── terraform.tfvars   ← Dev-specific values<br>│       └── backend.hcl        ← State storage location<br>└── modules/<br>    ├── vpc/             ← Network (3-tier architecture)<br>    ├── eks/             ← Kubernetes cluster + IRSA<br>    ├── rds/             ← PostgreSQL database<br>    ├── dynamodb/        ← NoSQL tables<br>    ├── elasticache/     ← Redis cache<br>    ├── messaging/       ← SNS topics + SQS queues<br>    ├── ecr/             ← Container image repos<br>    ├── s3/              ← Object storage<br>    ├── vpc-endpoints/   ← Private AWS API access<br>    └── secrets/         ← Secrets Manager (Vault bootstrap)</pre><p><strong>Why this structure?</strong> Each module can be understood, tested, and modified independently. When you explain this in an interview, you can say: “I split infrastructure into 10 Terraform modules. The VPC module knows nothing about databases. The EKS module knows nothing about messaging. This means I can modify the database configuration without any risk of accidentally changing the network.”</p><h3>Step 2: Bootstrap Terraform State</h3><p>Terraform state tracks what infrastructure exists. It must be stored remotely so multiple engineers can collaborate, and so it survives if your laptop dies.</p><pre>chmod +x scripts/bootstrap-state.sh<br><br>./scripts/bootstrap-state.sh dev</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*0XjXaKCG9Uwlas6DgVRMFA.png" /></figure><p>This creates:</p><ul><li><strong>S3 bucket</strong> (jobboard-dev-terraform-state) — stores the state file, versioned so you can recover from corruption</li><li><strong>DynamoDB table</strong> (jobboard-dev-terraform-locks) — prevents two people from running terraform apply simultaneously</li></ul><p><strong>Why DynamoDB locking?</strong> Without it, two engineers running terraform apply at the same time can corrupt the state file. The DynamoDB lock is atomic — only one process can hold it.</p><h3>Step 3: Terraform Init + Plan</h3><pre>cd terraform<br><br># Initialise - downloads providers, configures state backend<br>terraform init -backend-config=environments/dev/backend.hcl<br><br># Plan - shows what Terraform WILL create (doesn&#39;t create anything yet)<br>terraform plan -var-file=environments/dev/terraform.tfvars -out=plan.out</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*NTrVvKnFw9YV37W5p_P48g.png" /></figure><p><strong>Always review the plan.</strong> You should see approximately 70–80 resources. The key ones to verify:</p><p>Count Resource Type Purpose 1 VPC Network boundary 9 Subnets 3 public + 3 private + 3 isolated 1 Internet Gateway Public internet access 1 NAT Gateway + EIP Outbound for private subnets 3 Route Tables One per subnet tier 1 EKS Cluster Kubernetes control plane 1 Node Group EC2 worker instances 1 OIDC Provider Enables IRSA 4 IRSA Roles Per-service IAM roles 1 RDS Instance PostgreSQL database 3 DynamoDB Tables Applications, notifications, preferences 1 ElastiCache Cluster Redis cache 3 SNS Topics Event publishing 6 SQS Queues 3 main + 3 DLQs 5 ECR Repositories Container image storage 9 VPC Endpoints Private AWS API access</p><h3>Step 4: Terraform Apply</h3><pre>terraform apply plan.out</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*wetlKyoR7dvHgBL0MHfpsQ.png" /></figure><p><strong>This takes approximately 15–20 minutes.</strong> The longest resources:</p><ul><li>EKS cluster: ~10 minutes (AWS provisions the control plane)</li><li>RDS instance: ~5 minutes (database initialisation)</li><li>VPC endpoints: ~3 minutes each</li></ul><p>While waiting, let’s understand what’s being created and why.</p><h3>The Security Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1nkvn_lrUjppl-evwo4RQ.png" /></figure><h3><strong>The VPC is a three-tier architecture:</strong></h3><pre>VPC (10.0.0.0/16) — 65,536 IP addresses<br>│<br>├── Public Subnets (10.0.1-3.0/24)<br>│   ├── Application Load Balancer (receives internet traffic)<br>│   └── NAT Gateway (outbound internet for private subnets)<br>│<br>├── Private Subnets (10.0.10-12.0/24)<br>│   └── EKS Worker Nodes (run our pods)<br>│<br>└── Isolated Subnets (10.0.20-22.0/24)<br>    ├── RDS PostgreSQL (no internet route at all)<br>    └── ElastiCache Redis (no internet route at all)</pre><p><strong>The critical insight:</strong> It’s the route table that makes a subnet public, private, or isolated:</p><ul><li>Public route table: 0.0.0.0/0 → Internet Gateway = full internet access</li><li>Private route table: 0.0.0.0/0 → NAT Gateway = outbound only</li><li>Isolated route table: no default route = only VPC-internal traffic</li></ul><p><strong>Interview answer:</strong> “I use a three-tier VPC architecture. Isolated subnets for databases have no internet route — even if the database is compromised, data can’t be exfiltrated externally. This is defence-in-depth.”</p><h3>IRSA (IAM Roles for Service Accounts)</h3><p>This is the most important EKS security feature. Each microservice gets its own IAM role:</p><ul><li><strong>User Service:</strong> Can publish to SNS only</li><li><strong>Job Service:</strong> Can publish to SNS only</li><li><strong>Application Service:</strong> Can access DynamoDB + SNS + S3 + Vault Transit</li><li><strong>Notification Service:</strong> Can access DynamoDB + SQS + SES</li></ul><p>No service can access resources it doesn’t need. The Notification Service physically cannot reach the database — its IAM policy doesn’t allow it.</p><p><strong>How it works:</strong> When a pod’s AWS SDK makes an API call, it automatically gets temporary credentials from the EKS OIDC provider. No access keys stored anywhere.</p><h3>VPC Endpoints</h3><p>Without endpoints, every call to DynamoDB, SQS, SNS, etc., goes through the NAT Gateway. NAT Gateway charges £0.035/GB for data processing. With endpoints, that traffic stays on the private AWS network — free and more secure.</p><h3><strong>Data Flow Architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6wX_6pvOVttYDGVbBv7fQ.png" /></figure><h3>Step 5: Configure kubectl</h3><p>Once Terraform completes:</p><pre># Configure kubectl to talk to your cluster<br>aws eks update-kubeconfig --region eu-west-2 --name jobboard-dev-eks<br><br># Verify nodes are ready<br>kubectl get nodes<br># NAME                                           STATUS   ROLES    AGE   VERSION<br># ip-10-0-10-xxx.eu-west-2.compute.internal      Ready    &lt;none&gt;   5m    v1.29.x<br># ip-10-0-11-xxx.eu-west-2.compute.internal      Ready    &lt;none&gt;   5m    v1.29.x</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*StIenoNmA7frSqNwHEl-6w.png" /></figure><p>Create the application namespace:</p><pre>kubectl create namespace jobboard</pre><h3>Step 6: Deploy Vault</h3><p>Vault is the primary secrets engine. It replaces static Kubernetes Secrets with dynamic, short-lived credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tComeqNQKNrEWrka0PM6uA.png" /></figure><h3>Install Vault via Helm</h3><pre># Create Vault namespace with label (needed for network policies)<br>kubectl create namespace vault<br>kubectl label namespace vault name=vault<br># Add HashiCorp Helm repo<br>helm repo add hashicorp https://helm.releases.hashicorp.com<br>helm repo update<br># Install Vault in dev mode (auto-unsealed, in-memory storage)<br># Production would use HA mode with Raft storage + KMS auto-unseal<br>helm install vault hashicorp/vault \<br>  --namespace vault \<br>  --set &quot;server.dev.enabled=true&quot; \<br>  --set &quot;server.dev.devRootToken=root&quot; \<br>  --set &quot;injector.enabled=true&quot; \<br>  --set &quot;injector.replicas=1&quot;<br># Wait for Vault to be ready<br>kubectl -n vault wait --for=condition=ready pod/vault-0 --timeout=120s</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*m3J9KcN5iw6DVeIRe5BaIQ.png" /></figure><p><strong>What we just deployed:</strong></p><ul><li><strong>vault-0 pod:</strong> The Vault server that stores and manages secrets</li><li><strong>vault-agent-injector:</strong> A Kubernetes mutating webhook that watches for pods with Vault annotations and adds sidecar containers that fetch secrets</li></ul><h3>Configure Vault</h3><p>Run the setup script, passing your RDS and Redis endpoints:</p><pre># Get the endpoints from Terraform outputs<br>cd terraform<br>RDS_ENDPOINT=$(terraform output -raw rds_endpoint)<br>RDS_PASSWORD=$(terraform output -raw rds_master_password 2&gt;/dev/null || \<br>  aws secretsmanager get-secret-value --secret-id jobboard-dev/rds/master \<br>  --query &#39;SecretString&#39; --output text | jq -r &#39;.password&#39;)<br>REDIS_ENDPOINT=$(terraform output -raw redis_endpoint)<br>cd ..<br># Run the Vault setup<br>chmod +x scripts/setup-vault.sh<br>./scripts/setup-vault.sh &quot;$RDS_ENDPOINT&quot; &quot;$RDS_PASSWORD&quot; &quot;$REDIS_ENDPOINT&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*dd6QywX0GnLy2-DLIJ1gZg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*B5mssn1q_z9KG8kG3ue_og.png" /></figure><p><strong>What this configures:</strong></p><ol><li><strong>Kubernetes Auth</strong> — Vault validates pod identity by checking ServiceAccount tokens against the Kubernetes API. When a user-service pod requests a secret, Vault confirms: “Is this actually the user-service ServiceAccount in the jobboard namespace?”</li><li><strong>Database Secrets Engine</strong> — Instead of a static database password, Vault generates a unique PostgreSQL user for each pod. The credentials have a 1-hour TTL and are automatically revoked when the pod terminates.</li><li><strong>KV v2 Static Secrets</strong> — JWT signing keys, Redis connection strings, bcrypt rounds. Stored with versioning so accidental overwrites can be rolled back.</li><li><strong>Transit Engine</strong> — Encryption-as-a-service for PII. The Application Service sends plaintext to Vault, gets back ciphertext, and stores it in DynamoDB. The encryption key never leaves Vault.</li><li><strong>Per-service Policies</strong> — Each service can only access its own secrets. The user-service policy can’t read job-service secrets. The notification-service policy can’t use the Transit engine.</li><li><strong>Audit Logging</strong> — All secret accesses are logged. Vault is fail-closed: if audit logging fails, Vault stops serving secrets rather than continuing to operate without an audit trail.</li></ol><h3>Verify Vault</h3><pre># Test dynamic credential generation<br>kubectl -n vault exec vault-0 -- vault read database/creds/user-service<br># Key              Value<br># lease_id         database/creds/user-service/abc123...<br># lease_duration   1h<br># username         v-k8s-user-se-xyz789<br># password         A1b2C3d4-random-password<br><br># Each call generates NEW unique credentials<br>kubectl -n vault exec vault-0 -- vault read database/creds/user-service<br># username will be DIFFERENT from above</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*kGWHWuV5GuUl2RX2WC5uug.png" /></figure><p><strong>Interview gold:</strong> “Each pod gets unique database credentials with a 1-hour TTL. If a pod is compromised, the attacker has at most 1 hour of access with limited permissions. When the pod terminates, Vault revokes the credentials and drops the PostgreSQL role entirely.”</p><h3>Step 7: Build and Push Container Images</h3><pre># Get your AWS account ID<br>ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)<br>REGION=&quot;eu-west-2&quot;<br>REGISTRY=&quot;${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com&quot;<br># Authenticate Docker to ECR<br>aws ecr get-login-password --region ${REGION} | \<br>  docker login --username AWS --password-stdin ${REGISTRY}<br># Build and push each service<br>for service in user-service job-service application-service notification-service; do<br>  echo &quot;Building ${service}...&quot;<br>  docker build -t ${REGISTRY}/jobboard-dev/${service}:v1 ./services/${service}/<br>  docker push ${REGISTRY}/jobboard-dev/${service}:v1<br>  echo &quot;Pushed ${service}:v1&quot;<br>done</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*CvftdXz4QQ4UHTT3Eljf7w.png" /></figure><p>Each service uses a <strong>slim Dockerfile</strong> that installs dependencies, creates a non-root user (app with UID 1000), and sets a health check on /health.</p><h3>Step 8: Deploy Services with Helm</h3><p>Before deploying, update the Helm values files with your actual endpoints:</p><pre>ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)<br>RDS_ENDPOINT=$(cd terraform &amp;&amp; terraform output -raw rds_endpoint &amp;&amp; cd ..)<br>REDIS_ENDPOINT=$(cd terraform &amp;&amp; terraform output -raw redis_endpoint &amp;&amp; cd ..)<br><br># Update placeholders in all values files<br>for file in helm-charts/microservice/values/*.yaml; do<br>  sed -i &#39;&#39; &quot;s/ACCOUNT_ID/${ACCOUNT_ID}/g&quot; &quot;$file&quot;<br>  sed -i &#39;&#39; &quot;s/RDS_ENDPOINT/${RDS_ENDPOINT}/g&quot; &quot;$file&quot;<br>  sed -i &#39;&#39; &quot;s/REDIS_ENDPOINT/${REDIS_ENDPOINT}/g&quot; &quot;$file&quot;<br>done</pre><p>Deploy each service:</p><pre># Install Metrics Server (required for HPA)<br>kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml<br><br># Deploy each service<br>for service in user-service job-service application-service notification-service; do<br>  helm upgrade --install ${service} ./helm-charts/microservice \<br>    -f helm-charts/microservice/values/${service}.yaml \<br>    -n jobboard<br>  echo &quot;Deployed ${service}&quot;<br>done<br># Check pods - should show 2/2 (app container + vault-agent sidecar)<br>kubectl -n jobboard get pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*918aY8EmQDlkMXaDGH0TTw.png" /></figure><p>Each pod runs with <strong>2 containers</strong>:</p><ol><li>The application container (your code)</li><li>The Vault Agent sidecar (authenticates, fetches secrets, renews leases)</li></ol><p>The Vault Agent Injector (webhook) automatically adds the sidecar when it sees the vault.hashicorp.com/agent-inject: &quot;true&quot; annotation in the deployment template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*LrVFwsMhkOjvLrxI-xoRUQ.png" /></figure><h4><strong>Initialise Database Schema</strong></h4><p>Vault dynamic credentials only have DML permissions (SELECT, INSERT, UPDATE, DELETE). They deliberately cannot CREATE or DROP tables — this is least-privilege by design. If a pod is compromised, the attacker can read and write data but cannot alter the schema, drop tables, or escalate access.</p><pre># The master user (stored in Secrets Manager) runs schema migrations once:<br><br>RDS_ENDPOINT=$(cd terraform &amp;&amp; terraform output -raw rds_endpoint &amp;&amp; cd ..)<br>RDS_PASSWORD=$(cd terraform &amp;&amp; terraform output -raw rds_master_password)<br><br>kubectl run db-init -n jobboard --image=postgres:16 --rm -i --restart=Never \<br>  --env=&quot;PGPASSWORD=$RDS_PASSWORD&quot; -- \<br>  psql &quot;postgresql://jobboard_admin@${RDS_ENDPOINT}/jobboard?sslmode=require&quot; -c &quot;<br>    CREATE TABLE IF NOT EXISTS users (<br>      id SERIAL PRIMARY KEY,<br>      email VARCHAR(255) UNIQUE NOT NULL,<br>      password_hash VARCHAR(255) NOT NULL,<br>      first_name VARCHAR(100) NOT NULL,<br>      last_name VARCHAR(100) NOT NULL,<br>      role VARCHAR(20) NOT NULL DEFAULT &#39;candidate&#39;,<br>      created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,<br>      updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP<br>    );<br><br>    CREATE TABLE IF NOT EXISTS jobs (<br>      id SERIAL PRIMARY KEY,<br>      title VARCHAR(255) NOT NULL,<br>      description TEXT,<br>      location VARCHAR(255),<br>      job_type VARCHAR(50) DEFAULT &#39;full-time&#39;,<br>      category VARCHAR(100),<br>      salary_min INTEGER,<br>      salary_max INTEGER,<br>      employer_id INTEGER,<br>      status VARCHAR(20) DEFAULT &#39;active&#39;,<br>      created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,<br>      updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP<br>    );<br>  &quot;</pre><p>This separation — master user owns the schema, Vault users own the data — mirrors how production teams operate. In a CI/CD pipeline, schema migrations would run as a Kubernetes Job using the master credentials, while application pods never have DDL access.</p><p><strong>Interview answer:</strong> “Schema migrations run separately using the master database user. Application pods only get SELECT, INSERT, UPDATE, and DELETE through Vault dynamic credentials. This means even if a pod is compromised, the attacker cannot DROP or ALTER tables — they’re limited to row-level operations with credentials that expire in one hour.”</p><h3>Step 9: Install AWS Load Balancer Controller</h3><p>Instead of the retired Ingress NGINX Controller, we use the AWS Load Balancer Controller with path-based routing:</p><pre># Install AWS Load Balancer Controller<br>helm repo add eks https://aws.github.io/eks-charts<br>helm repo update<br><br>helm install aws-load-balancer-controller eks/aws-load-balancer-controller \<br>  -n kube-system \<br>  --set clusterName=jobboard-dev-eks \<br>  --set serviceAccount.create=true \<br>  --set serviceAccount.name=aws-load-balancer-controller</pre><p><strong>Why not NGINX Ingress?</strong> The Kubernetes Ingress NGINX Controller was retired in November 2025. The AWS Load Balancer Controller is actively maintained, creates actual AWS ALBs (with WAF integration, SSL termination via ACM, and access logging), and is the recommended approach for EKS.</p><h3>Step 10: Apply Network Policies</h3><pre># Install Network policies (VPC CNI enforcement)<br>aws eks create-addon \<br>  --cluster-name jobboard-dev-eks \<br>  --addon-name vpc-cni \<br>  --configuration-values &#39;{&quot;enableNetworkPolicy&quot;:&quot;true&quot;}&#39; \<br>  --resolve-conflicts OVERWRITE<br><br>aws eks wait addon-active --cluster-name jobboard-dev-eks --addon-name vpc-cni<br><br># Apply our policies<br>kubectl apply -f kubernetes/security/network-policies.yaml<br><br># Verify - this should SUCCEED (allowed by policy)<br>kubectl -n jobboard exec deployment/user-service -c user-service -- \<br>  wget -qO- --timeout=3 http://job-service:80/health || echo &quot;Connection tested&quot;<br><br># Verify - this should FAIL (notification-service cannot reach user-service)<br>kubectl -n jobboard exec deployment/notification-service -c notification-service -- \<br>  node -e &quot;fetch(&#39;http://user-service:80/health&#39;,{signal:AbortSignal.timeout(3000)}).then(r=&gt;console.log(&#39;OPEN&#39;)).catch(e=&gt;console.log(&#39;BLOCKED&#39;))&quot;<br># Should print: BLOCKED</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4yAQiTeUgEfx3vjXLeYqXQ.png" /></figure><p><strong>Health Probe Policy</strong></p><p>When network policy enforcement is enabled, there’s a critical rule that’s easy to miss: <strong>kubelet health probes are blocked by default-deny</strong>.</p><p>The kubelet runs on the node (within the VPC CIDR) and sends HTTP requests to each pod’s health and readiness endpoints. Without an explicit ingress rule allowing this, every pod fails its readiness probe and shows 1/2 ReadyThe Vault sidecar runs fine, but Kubernetes thinks the application container is unhealthy.</p><pre># Allow kubelet health probes from node IPs<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-health-probes<br>  namespace: jobboard<br>  labels:<br>    app.kubernetes.io/part-of: jobboard<br>    policy-type: baseline<br>spec:<br>  podSelector: {}<br>  policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        - ipBlock:<br>            cidr: 10.0.0.0/16<br>      ports:<br>        - protocol: TCP<br>          port: 5001<br>        - protocol: TCP<br>          port: 5002<br>        - protocol: TCP<br>          port: 5003<br>        - protocol: TCP<br>          port: 5004</pre><p>This allows traffic from any IP in the VPC CIDR (10.0.0.0/16) to each service&#39;s port. Since the kubelet runs on the node (which has a 10.0.x.x address), the health probes pass. External traffic still can&#39;t reach these ports because they&#39;re in private subnets behind the VPC boundary.</p><p><strong>What the policies enforce:</strong></p><ul><li>Default deny: nothing can talk to anything unless explicitly allowed</li><li>DNS (port 53): all pods need name resolution</li><li>Vault (port 8200): all pods need secrets</li><li>AWS APIs (port 443): all pods need AWS service access via VPC endpoints</li><li>Per-service: only allowed communication paths (API gateway → services, application-service → job-service for validation)</li></ul><h3>Step 11: Install OPA Gatekeeper</h3><pre># Apply templates FIRST<br>kubectl apply -f kubernetes/security/gatekeeper-templates.yaml<br><br># Wait for CRDs to register<br>sleep 30<br><br># THEN apply constraints<br>kubectl apply -f kubernetes/security/gatekeeper-constraints.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pxQ72NSoVSpVi-C__AU1uw.png" /></figure><p>Test that it works:</p><pre># This should be REJECTED — no resource limits<br>kubectl -n jobboard run test-no-limits --image=nginx<br># Error from server: admission webhook &quot;validation.gatekeeper.sh&quot; denied the request<br><br># This should be REJECTED - not from approved registry<br>kubectl -n jobboard run test-bad-registry --image=nginx --requests=&#39;cpu=100m,memory=128Mi&#39; --limits=&#39;cpu=200m,memory=256Mi&#39;<br># Error: Container uses disallowed image</pre><h3>Step 12: Verify Horizontal Pod Autoscaler</h3><pre>kubectl -n jobboard get hpa<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jv0CgJdvs6yY4Bba1JzfPw.png" /></figure><p>The HPA monitors CPU utilisation and scales pods within the defined range. Note that notification-service has a lower threshold (60%) because it’s I/O bound from SQS polling — by the time CPU spikes, the queue is already backing up.</p><p><strong>HPA scale-down behaviour:</strong> The stabilizationWindowSeconds: 300 In the Helm template, HPA waits 5 minutes before scaling down. This prevents thrashing — a brief traffic spike shouldn&#39;t trigger scale-up, then immediate scale-down.</p><h3>Step 13: End-to-End Verification</h3><p>Port-forward to test the full chain:</p><pre>kubectl -n jobboard port-forward svc/user-service 5001:80 &amp;<br>kubectl -n jobboard port-forward svc/job-service 5002:80 &amp;<br><br># Register a user<br>curl -s -X POST http://localhost:5001/api/v1/auth/register \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;email&quot;:&quot;test@example.com&quot;,&quot;password&quot;:&quot;SecurePass123&quot;,&quot;first_name&quot;:&quot;Test&quot;,&quot;last_name&quot;:&quot;User&quot;,&quot;role&quot;:&quot;employer&quot;}&#39; | jq .<br># Login and get JWT token<br>TOKEN=$(curl -s -X POST http://localhost:5001/api/v1/auth/login \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;email&quot;:&quot;test@example.com&quot;,&quot;password&quot;:&quot;SecurePass123&quot;}&#39; | jq -r &#39;.token&#39;)<br># Create a job listing<br>curl -s -X POST http://localhost:5002/api/v1/jobs \<br>  -H &quot;Authorization: Bearer ${TOKEN}&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;title&quot;:&quot;Senior DevOps Engineer&quot;,&quot;description&quot;:&quot;Build cloud platforms&quot;,&quot;location&quot;:&quot;London&quot;,&quot;job_type&quot;:&quot;full-time&quot;,&quot;category&quot;:&quot;Engineering&quot;}&#39; | jq .<br># Search jobs<br>curl -s http://localhost:5002/api/v1/jobs | jq .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*gC1k5MiuP2MMekd6vzK6UA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*yaWaFVH8kn3FAOjGAM0qCg.png" /></figure><p><strong>Verify Vault dynamic credentials are working:</strong></p><pre># Check each pod has unique database credentials<br>kubectl -n jobboard exec deployment/user-service -c vault-agent -- \<br>  cat /vault/secrets/db-creds<br># Should show: postgresql://v-k8s-user-se-UNIQUE@rds-endpoint:5432/jobboard<br><br>kubectl -n jobboard exec deployment/job-service -c vault-agent -- \<br>  cat /vault/secrets/db-creds<br># Should show a DIFFERENT username than above</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*8XMx-OqwdT2N0Va3u3SEoQ.png" /></figure><h3>Cleanup</h3><pre><br># Destroy everything<br>terraform destroy -var-file=environments/dev/terraform.tfvars -auto-approve<br></pre><p>The state file in S3 remembers everything. terraform apply recreates the identical infrastructure. Your Helm charts and application code are in Git — redeploy takes 5 minutes after infrastructure is up.</p><h3>What You’ve Built: The Complete Picture</h3><pre>Internet → AWS ALB (Load Balancer Controller)<br>              ├── /api/v1/auth/*          → User Service (2-8 pods)<br>              ├── /api/v1/users/*         → User Service<br>              ├── /api/v1/jobs/*          → Job Service (2-10 pods)<br>              ├── /api/v1/applications/*  → Application Service (2-8 pods)<br>              └── /api/v1/notifications/* → Notification Service (2-6 pods)<br>Secrets: Vault Agent Sidecar → /vault/secrets/<br>  ├── Dynamic DB credentials (unique per pod, 1h TTL, auto-revoked)<br>  ├── Static secrets (JWT keys, Redis URL, config)<br>  └── Transit encryption (PII encrypted via Vault API)<br>Security Layers:<br>  1. VPC: 3-tier network (public/private/isolated)<br>  2. Security Groups: port-level access control<br>  3. IRSA: per-service AWS permissions (no access keys)<br>  4. Vault: per-service secret access policies<br>  5. Network Policies: default-deny + explicit allow<br>  6. OPA Gatekeeper: admission control (block insecure deployments)<br>  7. Pod Security Context: non-root, read-only FS, drop capabilities<br>  8. KMS: etcd encryption at rest<br>Scaling: HPA per service with tuned thresholds</pre><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*MhKmwX3MRt8v3EKjgISR3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8HUNToed3NAzzB-WAr-u3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*qqPvofAbdnA3Eyfi21ZV3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*uyfKhQnrS6UHv2Z_87TyLg.png" /></figure><h3>What’s Next: Part 2</h3><p>In the next article, we’ll add:</p><ul><li><strong>GitLab CI/CD pipelines</strong> with reusable templates</li><li><strong>Infrastructure pipeline</strong> for Terraform plan/apply</li><li><strong>ArgoCD GitOps</strong> with ApplicationSets</li><li><strong>Trivy</strong> container vulnerability scanning as a pipeline gate</li><li><strong>GitLab SAST</strong> and secret detection</li></ul><h3>Interview Preparation</h3><p>Every component in this platform answers an interview question. Here are the most likely ones:</p><p><strong>“Describe your experience with Vault.”</strong> “I deployed Vault on EKS with three secret engines: the database engine generates unique PostgreSQL credentials per pod with 1-hour TTLs, KV v2 stores static secrets like JWT keys, and the Transit engine provides encryption-as-a-service for PII. Vault Agent Injector adds a sidecar to each pod that handles authentication and secret retrieval — the application just reads files from a volume.”</p><p><strong>“How do you secure a Kubernetes platform?”</strong> “Defence in depth with seven layers: three-tier VPC for network isolation, security groups for port-level control, IRSA for zero-credential AWS access, Vault for dynamic secrets, VPC CNI network policies for microsegmentation, OPA Gatekeeper for admission control, and pod security contexts for container hardening.”</p><p><strong>“How do you handle database credentials?”</strong> “Vault’s database secrets engine generates unique, short-lived PostgreSQL credentials for each pod. TTL is 1 hour, renewal is automatic via the Vault Agent sidecar, and credentials are revoked when the pod terminates. This eliminates shared passwords.”</p><p><strong>“Why multiple databases?”</strong> “Polyglot persistence — each database is chosen for its access pattern. PostgreSQL for users and jobs because the data is relational, uses foreign keys, and requires full-text search. DynamoDB for applications because the schema varies per employer, and access patterns map cleanly to GSIs. Redis for job caching because the read-to-write ratio is 100:1.”</p><p><em>Tobi Akinlade is a DevOps Engineer specialising in cloud platform engineering, Infrastructure as Code, and Kubernetes. Connect on </em><a href="https://linkedin.com/in/tobiakinlade"><em>LinkedIn</em></a><em>.</em></p><p><strong>Tags:</strong> #DevOps #AWS #EKS #Terraform #Vault #Kubernetes #Microservices #PlatformEngineering #InfrastructureAsCode #CloudSecurity</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18f6b39630ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Zero to Production on AWS: A DevOps Series — Part 1: Infrastructure as Code]]></title>
            <link>https://medium.com/@tobi.akinlade.co/from-zero-to-production-on-aws-a-devops-series-part-1-infrastructure-as-code-b2ad7536b54f?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2ad7536b54f</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 15:18:58 GMT</pubDate>
            <atom:updated>2026-01-29T15:18:58.627Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Zero to Production on AWS: A DevOps Series — Part 1: Infrastructure as Code</h3><p><strong>How I architected and deployed a secure, scalable enterprise platform using Infrastructure as Code</strong></p><p>Modern enterprise IT demands infrastructure that’s secure, scalable, and cost-effective. When building a service management platform for internal operations, I set clear targets: 99.9% availability, defense-in-depth security, auto-scaling capability, and a development environment budget under £100 per month.</p><p>This article walks through the architecture, implementation, and a complete step-by-step deployment guide. While cloud-native approaches like Kubernetes dominate tech headlines, traditional infrastructure patterns remain the backbone of government and enterprise IT — and for good reason.</p><p><strong>Complete codebase on GitHub:</strong> <a href="https://github.com/tobiakinlade/service-hub-project.git">https://github.com/tobiakinlade/service-hub-project.git</a></p><h3>What We Built</h3><p>The platform implements a classic three-tier architecture that, despite being decades old, remains highly relevant: clear separation of concerns, straightforward security boundaries, and predictable scaling characteristics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqW9keJ5UF_DZfvN5YPhZg.png" /></figure><h3>The Three Tiers</h3><p><strong>Presentation Layer (Public Subnets):</strong> An Application Load Balancer spanning two availability zones handles incoming requests, SSL/TLS termination, and health-based traffic distribution.</p><p><strong>Application Layer (Private Subnets):</strong> EC2 instances in Auto Scaling Groups with no direct internet access. Outbound connectivity flows through NAT Gateways — isolated from direct public exposure.</p><p><strong>Data Layer (Private Database Subnets):</strong> RDS PostgreSQL and ElastiCache Redis with complete network isolation. Access is permitted only from the application tier.</p><h3>Key Design Decisions</h3><p><strong>Multi-AZ Deployment:</strong> Each tier spans two availability zones (eu-west-2a and eu-west-2b), providing resilience against datacenter failures while keeping development costs reasonable.</p><p><strong>Private Subnet Architecture:</strong> Application and database tiers exist entirely in private subnets. This defense-in-depth approach means even if an application vulnerability is exploited, the attacker gains no direct internet access and cannot reach the database directly.</p><p><strong>NAT Gateway Strategy:</strong> Two NAT Gateways (one per AZ) for high availability. For cost optimization, this can be reduced to one with accepted downtime risk.</p><h3>Network Design</h3><p>The network uses carefully planned CIDR blocks that allow for future expansion while maintaining clear segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZFfgu04is9r0ClUiIuiew.png" /></figure><h3>CIDR Planning</h3><p>The /16 VPC provides 65,536 IP addresses — far more than needed but allowing headroom for growth:</p><p>Subnet TypeCIDR RangePurposePublic10.0.1.0/24, 10.0.2.0/24Load balancers, NAT GatewaysApp Private10.0.11.0/24, 10.0.12.0/24Application instancesDB Private10.0.21.0/24, 10.0.22.0/24Databases (isolated)</p><p>The numbering scheme (1.x for public, 11.x for app, 21.x for database) makes subnet purpose immediately obvious when troubleshooting.</p><h3>Routing Strategy</h3><p>SubnetInternet AccessRoutePublicDirect (inbound + outbound)0.0.0.0/0 → Internet GatewayApp PrivateOutbound only0.0.0.0/0 → NAT GatewayDB Private<strong>None</strong>No internet route</p><p>Database subnets have no route to the internet whatsoever — complete isolation.</p><h3>Security: Defense in Depth</h3><p>Security isn’t a checkbox — it’s a layered approach where multiple defensive mechanisms work together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Si7WAAnT79oLJiffdzqL9Q.png" /></figure><h3>Security Group Chain</h3><pre>Internet → ALB (Port 80/443)<br>              ↓<br>           EC2 (Port 5000) ← Only from ALB<br>              ↓<br>           RDS (Port 5432) ← Only from EC2<br>           Redis (Port 6379) ← Only from EC2</pre><p><strong>Key Principle:</strong> Every rule references another security group, not IP ranges. If an application instance is compromised, the attacker can only communicate with resources the application legitimately needs.</p><h3>IAM Roles Over Access Keys</h3><p>EC2 instances use IAM roles — never access keys. The role grants exactly three S3 operations on exactly one bucket:</p><ul><li>s3:PutObject — Upload attachments</li><li>s3:GetObject — Retrieve attachments</li><li>s3:DeleteObject — Remove attachments</li></ul><p>Nothing more. If compromised, attackers cannot access other AWS resources.</p><h3>Step-by-Step Deployment Guide</h3><h3>Prerequisites</h3><p>Before starting, ensure you have:</p><ul><li>AWS account with admin permissions</li><li>AWS CLI installed and configured (aws configure)</li><li>Terraform 1.5+ installed</li><li>Git installed</li></ul><h3>Project Structure</h3><pre>servicehub-project/<br>├── terraform-infrastructure/<br>│   ├── modules/<br>│   │   ├── vpc/           # Network foundation<br>│   │   ├── security-groups/  # Firewall rules<br>│   │   ├── iam/           # Roles and policies<br>│   │   ├── s3/            # Object storage<br>│   │   ├── alb/           # Load balancer<br>│   │   ├── ec2/           # Compute (ASG + Launch Template)<br>│   │   ├── rds/           # PostgreSQL database<br>│   │   └── elasticache/   # Redis cache<br>│   └── environments/<br>│       └── dev/<br>│           ├── main.tf<br>│           ├── variables.tf<br>│           ├── outputs.tf<br>│           └── user_data.sh</pre><h3>Step 1: Clone and Setup</h3><pre>git clone https://github.com/tobiakinlade/service-hub-project.git<br><br>cd service-hub-project/terraform/environments/dev</pre><h3>Step 2: Create S3 Backend for State</h3><p>Terraform state should be stored remotely for safety and team collaboration.</p><p><strong>Key concept — backend configuration:</strong></p><pre>terraform {<br>  backend &quot;s3&quot; {<br>    bucket         = &quot;servicehub-terraform-state-ACCOUNT_ID&quot;<br>    key            = &quot;dev/terraform.tfstate&quot;<br>    region         = &quot;eu-west-2&quot;<br>    dynamodb_table = &quot;servicehub-terraform-locks&quot;<br>    encrypt        = true<br>  }<br>}</pre><p><strong>Why this matters:</strong></p><ul><li>S3 stores state with versioning (rollback capability)</li><li>DynamoDB provides locking (prevents concurrent modifications)</li><li>Encryption protects sensitive values in state</li></ul><p><strong>Run the setup script:</strong></p><pre>chmod +x setup-backend.sh<br><br>./setup-backend.sh</pre><h3>Step 3: Create EC2 Key Pair &amp; Get AMI ID</h3><p>Required for SSH access (emergency debugging only — SSM is preferred).</p><pre>aws ssm get-parameter \<br>  --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \<br>  --query &#39;Parameter.Value&#39; \<br>  --output text \<br>  --region eu-west-2<br><br>aws ec2 create-key-pair \<br>  --key-name servicehub-dev-key \<br>  --query &#39;KeyMaterial&#39; \<br>  --output text \<br>  --region eu-west-2 &gt; servicehub-dev-key.pem<br><br>chmod 400 servicehub-dev-key.pem</pre><h3>Step 4: Configure Variables</h3><p>Create terraform.tfvars with your settings:</p><pre>aws_region   = &quot;eu-west-2&quot;<br>project_name = &quot;servicehub&quot;<br>environment  = &quot;dev&quot;<br>vpc_cidr     = &quot;10.0.0.0/16&quot;<br>ec2_ami_id = &quot;ami-0737d2d50c7fece1b&quot;<br><br><br>ec2_key_name = &quot;servicehub-dev-key&quot;<br><br>common_tags = {<br>  Project     = &quot;ServiceHub&quot;<br>  Environment = &quot;dev&quot;<br>  ManagedBy   = &quot;Terraform&quot;<br>  Owner       = &quot;Tobi&quot;<br>}</pre><p>Set the database password as an environment variable (keeps it out of files):</p><pre>export TF_VAR_db_password=&#39;ServiceHubSecurePassword123!&#39;</pre><h3>Step 5: Understand the Module Structure</h3><p>Each Terraform module handles one concern. Here’s what each does:</p><p><strong>VPC Module</strong> — Creates the network foundation:</p><pre>module &quot;vpc&quot; {<br>  source = &quot;../../modules/vpc&quot;<br>  <br>  vpc_cidr           = &quot;10.0.0.0/16&quot;<br>  availability_zones = [&quot;eu-west-2a&quot;, &quot;eu-west-2b&quot;]<br>  <br>  # Subnet CIDRs follow the naming convention<br>  public_subnet_cidrs     = [&quot;10.0.1.0/24&quot;, &quot;10.0.2.0/24&quot;]<br>  private_app_subnet_cidrs = [&quot;10.0.11.0/24&quot;, &quot;10.0.12.0/24&quot;]<br>  private_db_subnet_cidrs  = [&quot;10.0.21.0/24&quot;, &quot;10.0.22.0/24&quot;]<br>  <br>  enable_nat_gateway = true<br>  enable_flow_logs   = true<br>}</pre><p><strong>Security Groups Module</strong> — Implements the security chain:</p><pre>module &quot;security_groups&quot; {<br>  source = &quot;../../modules/security-groups&quot;<br>  <br>  vpc_id   = module.vpc.vpc_id<br>  vpc_cidr = module.vpc.vpc_cidr<br>  <br>  # Creates: alb_sg, ec2_sg, rds_sg, redis_sg<br>  # Each references only the security groups it needs<br>}</pre><p><strong>EC2 Module</strong> — Configures Auto Scaling:</p><pre>module &quot;ec2&quot; {<br>  source = &quot;../../modules/ec2&quot;<br>  <br>  subnet_ids         = module.vpc.private_app_subnet_ids<br>  security_group_ids = [module.security_groups.ec2_sg_id]<br>  target_group_arns  = [module.alb.target_group_arn]<br>  <br>  instance_type = &quot;t3.small&quot;<br>  min_size      = 1<br>  max_size      = 3<br>  desired_size  = 1<br>  <br>  # Bootstrap script transforms bare instance into app server<br>  user_data = templatefile(&quot;user_data.sh&quot;, {<br>    db_endpoint    = module.rds.db_endpoint<br>    redis_endpoint = module.elasticache.redis_endpoint<br>  })<br>}</pre><p><strong>RDS Module</strong> — PostgreSQL database:</p><pre>module &quot;rds&quot; {<br>  source = &quot;../../modules/rds&quot;<br>  <br>  subnet_ids         = module.vpc.private_db_subnet_ids<br>  security_group_ids = [module.security_groups.rds_sg_id]<br>  <br>  instance_class    = &quot;db.t3.micro&quot;<br>  allocated_storage = 20<br>  multi_az          = false  # true for production<br>  <br>  backup_retention_period = 7<br>}</pre><h3>Step 6: Understand User Data Automation</h3><p>The user_data.sh script is critical—it transforms a bare EC2 instance into a running application server.</p><p><strong>Key concept — complete automation:</strong></p><pre>#!/bin/bash<br>set -e<br><br># 1. Install system packages<br>dnf install -y python3.11 python3.11-pip redis6 postgresql15<br># 2. Install Python dependencies<br>python3.11 -m pip install flask gunicorn psycopg2-binary redis<br># 3. Create Flask application with health endpoints<br>cat &gt; /opt/servicehub/app.py &lt;&lt; &#39;EOF&#39;<br># Flask app with /health/health, /health/ready, /health/live<br># These endpoints are checked by the ALB every 30 seconds<br>EOF<br># 4. Create systemd service for automatic restart<br>cat &gt; /etc/systemd/system/servicehub.service &lt;&lt; &#39;EOF&#39;<br>[Service]<br>ExecStart=/usr/local/bin/gunicorn --bind 0.0.0.0:5000 app:app<br>Restart=always<br>EOF<br># 5. Start and verify<br>systemctl enable --now servicehub<br>curl http://localhost:5000/health/health</pre><p><strong>Why this matters:</strong></p><ul><li>Every instance is identical (no configuration drift)</li><li>Auto Scaling launches production-ready instances automatically</li><li>Failed instances are replaced without manual intervention</li></ul><h3>Step 7: Deploy Infrastructure</h3><pre># Initialize Terraform (downloads providers, configures backend)<br>terraform init<br><br># Preview changes<br>terraform plan<br># Deploy (takes 10-15 minutes)<br>terraform apply</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gkLkmAfCQNAJteyz5mx_gg.png" /></figure><p><strong>What gets created:</strong></p><ul><li>1 VPC with 6 subnets across 2 AZs</li><li>2 NAT Gateways with Elastic IPs</li><li>4 Security Groups</li><li>1 Application Load Balancer</li><li>1 Auto Scaling Group with Launch Template</li><li>1 RDS PostgreSQL instance</li><li>1 ElastiCache Redis cluster</li><li>1 S3 bucket for attachments</li><li>IAM roles and policies</li></ul><h3>Step 8: Verify Deployment</h3><p><strong>Get the ALB endpoint:</strong></p><pre>ALB_DNS=$(terraform output -raw alb_dns_name)<br><br>echo &quot;Your API: http://${ALB_DNS}&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOjIZjqh-E1sc2R1FYgDJA.png" /></figure><p><strong>Test health endpoints:</strong></p><pre># Should return {&quot;status&quot;: &quot;healthy&quot;, &quot;timestamp&quot;: ...}<br>curl http://${ALB_DNS}/health/health<br><br># Check all endpoints<br>curl http://${ALB_DNS}/health/ready<br>curl http://${ALB_DNS}/health/live<br>curl http://${ALB_DNS}/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gS_dRHM1u30_V1cbc8iTaQ.png" /></figure><p><strong>Verify target health in AWS Console:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZtmoQXSjDvUlswtwBN2Vog.png" /></figure><h3>Step 9: Clean Up</h3><pre># Destroy all resources<br><br>terraform destroy --auto-approve</pre><p><strong>Important:</strong> This deletes everything including the database. For production, enable deletion_protection = true on RDS.</p><h3>Results</h3><h3>Availability</h3><ul><li>Multi-AZ deployment ✅</li><li>Auto-healing through ASG ✅</li><li>Health check-based routing ✅</li><li>Target uptime: 99.9%</li></ul><h3>Security</h3><ul><li>Network isolation (private subnets) ✅</li><li>Encryption at rest and in transit ✅</li><li>IAM roles (no access keys) ✅</li><li>Least privilege security groups ✅</li></ul><h3>Automation</h3><ul><li>Deployment time: 15 minutes (previously 4+ hours manual)</li><li>Configuration drift: Zero</li><li>Manual steps required: Zero</li></ul><h3>Key Takeaways</h3><h3>Technical Lessons</h3><p><strong>Infrastructure as Code Enables Reproducibility:</strong> The same Terraform modules deploy dev, staging, and production with different variables.</p><p><strong>Security Must Be Architectural:</strong> Private subnets, security group boundaries, and IAM roles create defense in depth.</p><p><strong>Automation Reduces Risk:</strong> Automated user_data scripts mean every instance is identical and known-good.</p><p><strong>Cost Optimization Starts With Architecture:</strong> The biggest savings come from architectural decisions — not finding cheaper instance types.</p><h3>Process Lessons</h3><p><strong>Test Components Independently:</strong> The user_data debugging would have been faster if I’d tested it standalone first.</p><p><strong>Use Remote State From Day One:</strong> Even for personal projects, S3 backend prevents numerous headaches.</p><p><strong>Design for Complete Automation:</strong> Placeholders like “Phase 2” create deployment friction.</p><h3>Next Steps</h3><p>This forms the foundation for a complete DevOps implementation:</p><ul><li><strong>Phase 2:</strong> Configuration Management with Ansible (server hardening, CIS compliance)</li><li><strong>Phase 3:</strong> Secrets Management and CI/CD (GitHub Actions pipeline)</li><li><strong>Phase 4:</strong> Observability (Prometheus, Grafana, alerting)</li></ul><h3>Conclusion</h3><p>Building production-grade infrastructure requires balancing security, availability, cost, and maintainability. This implementation demonstrates that traditional architectural patterns, implemented with modern tooling like Terraform, create robust systems meeting enterprise requirements.</p><p><strong>Complete codebase on GitHub:</strong> <a href="https://github.com/tobiakinlade/service-hub-project.git">https://github.com/tobiakinlade/service-hub-project.git</a></p><p>Whether you’re building for startups, enterprises, or government agencies, these patterns provide a solid foundation. The technologies may evolve, but the principles — separation of concerns, defense in depth, automation, and cost consciousness — remain constant.</p><p><strong>About the Author</strong></p><p>I’m a DevOps Engineer specialising in cloud infrastructure, automation, and security. With experience across government and private sector deployments, I focus on building reliable, secure infrastructure meeting stringent compliance requirements while remaining cost-effective.</p><p><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">https://www.linkedin.com/in/tobi-akinlade-02b035150/</a></p><p><em>Tags: </em>#AWS #Terraform #InfrastructureAsCode #CloudComputing #SRE</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2ad7536b54f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production-Grade Kubernetes Platform: Part 2 — Complete DevOps Automation]]></title>
            <link>https://medium.com/@tobi.akinlade.co/production-grade-kubernetes-platform-part-2-complete-devops-automation-df71d3768911?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/df71d3768911</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Wed, 31 Dec 2025 13:16:25 GMT</pubDate>
            <atom:updated>2025-12-31T13:16:25.490Z</atom:updated>
            <content:encoded><![CDATA[<h3>Production-Grade Kubernetes Platform: Part 2 — Complete DevOps Automation</h3><h3>From GitOps to CI/CD, Observability to SRE — Building Enterprise-Ready Infrastructure</h3><h3>Introduction</h3><p>In <a href="https://medium.com/@tobi.akinlade.co/building-a-production-grade-application-with-hashicorp-vault-on-aws-eks-c9110af7c5fa"><strong>Part 1</strong></a>, we built a production-ready UK Tax Calculator application on AWS EKS with HashiCorp Vault for secrets management. We established a solid foundation with a 3-node Kubernetes cluster, PostgreSQL database, and a secure application stack.</p><p>In Part 2, we’ll transform this foundation into a complete, enterprise-grade DevOps platform. We’ll implement automation, observability, and reliability practices that meet production standards.</p><h3>What You’ll Build</h3><p>By the end of this comprehensive guide, you’ll have:</p><ol><li><strong>Complete GitOps Automation</strong> — ArgoCD for declarative, Git-driven deployments</li><li><strong>CI/CD Pipeline with Security Gates</strong> — Automated builds, quality checks, and security scanning</li><li><strong>Full Observability Stack</strong> — Prometheus, Grafana, and Loki for metrics and logs</li></ol><h3>The Complete Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ak5Kr9byHutu3Bnvy3B94Q.jpeg" /></figure><h3>Prerequisites</h3><p><strong>GitHub:</strong> <a href="https://github.com/tobiakinlade/vault-gitops-platform.git">https://github.com/tobiakinlade/vault-gitops-platform.git</a></p><p>Before starting, ensure you have:</p><h3>Required</h3><ul><li>✅ Part 1 deployed and running (3-node EKS cluster with tax calculator application)</li><li>✅ kubectl installed and configured</li><li>✅ helm v3 installed</li><li>✅ AWS CLI configured</li><li>✅ GitHub account with a repository</li><li>✅ Basic knowledge of Kubernetes, Docker, and CI/CD</li></ul><h3>Verify Your Environment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*8lHqt5hH1O4RpomMYwdYmQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*-8vSk_4hlyNOrjTA4YW6PA.png" /></figure><h3>Architecture Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UyNZTQg8WHI1-lQwrUa_MA.jpeg" /></figure><h3>Part 1: Foundation Setup</h3><pre>## Project Structure Overview<br><br>Before we begin, let&#39;s understand how our project is organized. We use a clean separation of concerns:<br><br>- **`.github/workflows/`** - CI/CD automation pipelines<br>- **`argocd/applications/`** - GitOps application definitions  <br>- **`kubernetes/base/`** - Base Kubernetes manifests (Kustomize-style)<br>- **`tax-calculator-app/`** - Application source with security-hardened Dockerfiles<br>- **`helm-values/`** - Observability stack configurations<br><br>This structure ensures maintainability and follows GitOps best practices.</pre><h3>Step 1: Create Namespace Structure</h3><p>We’ll organise our cluster with dedicated namespaces for different operational domains.</p><pre># Create namespace configuration<br>cat &gt; namespaces.yaml &lt;&lt;&#39;EOF&#39;<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: argocd<br>  labels:<br>    name: argocd<br>    app.kubernetes.io/part-of: gitops<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: monitoring<br>  labels:<br>    name: monitoring<br>    app.kubernetes.io/part-of: observability<br>    pod-security.kubernetes.io/enforce: restricted<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: logging<br>  labels:<br>    name: logging<br>    app.kubernetes.io/part-of: observability<br>    pod-security.kubernetes.io/enforce: restricted<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: elastic-system<br>  labels:<br>    name: elastic-system<br>    app.kubernetes.io/part-of: logging<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: security<br>  labels:<br>    name: security<br>    app.kubernetes.io/part-of: security<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: cert-manager<br>  labels:<br>    name: cert-manager<br>    app.kubernetes.io/part-of: security<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: velero<br>  labels:<br>    name: velero<br>    app.kubernetes.io/part-of: operations<br>---<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: sonarqube<br>  labels:<br>    name: sonarqube<br>    app.kubernetes.io/part-of: cicd<br>EOF<br><br># Apply namespaces<br>kubectl apply -f namespaces.yaml<br># Verify creation<br>kubectl get namespaces</pre><p><strong>Expected Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*KRRC-4Vui3DXnDHZr3MS4w.png" /></figure><h3>Part 2: GitOps with ArgoCD</h3><blockquote><strong>GitOps = Git as Single Source of Truth</strong><br>Stop manual kubectl commands. Instead, declare your desired infrastructure state in Git (YAML files). ArgoCD continuously watches your repository and automatically syncs changes to the cluster. Benefits: Full audit trail in Git history, one-click rollbacks with git revert, and prevention of configuration drift. Every change is versioned, automated, and recoverable.</blockquote><h3>Step 2: Install ArgoCD</h3><pre># Install ArgoCD<br>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.9.3/manifests/install.yaml<br># Wait for pods to be ready (this takes 2-3 minutes)<br>echo &quot;Waiting for ArgoCD to be ready...&quot;<br>kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server -n argocd --timeout=300s<br># Check deployment<br>kubectl get pods -n argocd</pre><p><strong>Expected Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*zcJTgGQx2Gbg_kyQvMeHMQ.png" /></figure><h3>Step 3: Expose ArgoCD UI</h3><pre># Patch ArgoCD server service to LoadBalancer<br>kubectl patch svc argocd-server -n argocd -p &#39;{&quot;spec&quot;: {&quot;type&quot;: &quot;LoadBalancer&quot;}}&#39;<br># Wait for LoadBalancer to be provisioned<br>echo &quot;Waiting for LoadBalancer...&quot;<br>sleep 60<br># Get ArgoCD URL<br>ARGOCD_SERVER=$(kubectl get svc argocd-server -n argocd -o jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39;)<br>echo &quot;ArgoCD URL: https://${ARGOCD_SERVER}&quot;<br># Get initial admin password<br>ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=&quot;{.data.password}&quot; | base64 -d)<br># Save credentials<br>cat &gt; argocd-credentials.txt &lt;&lt;EOF<br>ArgoCD Access Information<br>========================<br>URL: https://${ARGOCD_SERVER}<br>Username: admin<br>Password: ${ARGOCD_PASSWORD}<br>IMPORTANT: Save this file securely and change the password after first login!<br>EOF<br>echo &quot;✓ ArgoCD credentials saved to argocd-credentials.txt&quot;<br>cat argocd-credentials.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*ZGu-aqsPNlDDpNdSpL3DRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*gty7qkce0YtRRLELPYbC_Q.png" /></figure><p><strong>Alternative Access (Port Forward):</strong></p><pre># If LoadBalancer is not available<br>kubectl port-forward svc/argocd-server -n argocd 8080:443<br># Access at: https://localhost:8080<br># Username: admin<br># Password: (from argocd-credentials.txt)</pre><h3>Step 4: Connect Your Git Repository</h3><pre># Add your Git repository using kubectl<br>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: vault-gitops-platform<br>  namespace: argocd<br>  labels:<br>    argocd.argoproj.io/secret-type: repository<br>stringData:<br>  url: https://github.com/tobiakinlade/vault-gitops-platform.git<br>  type: git<br>EOF<br><br># Verify it was added<br>kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*MQJ9GwOgbczaCC-B4AdcmQ.png" /></figure><h3>Part 3: CI/CD Pipeline with Security Scanning</h3><blockquote><strong>Defense-in-Depth Pipeline</strong><br>Modern CI/CD layers multiple security checks that every commit must pass:</blockquote><blockquote><strong>SAST</strong> (SonarQube): Scans source code for vulnerabilities before build</blockquote><blockquote><strong>Container Scanning</strong> (Trivy): Finds vulnerabilities in dependencies and OS packages</blockquote><blockquote><strong>Quality Gates</strong>: Enforces standards (80%+ coverage, duplication, and a security rating)</blockquote><blockquote><strong>Shift Left</strong>: Catch issues early — when they’re cheapest to fix. This pipeline transforms security from an audit to an automated gate.</blockquote><h3>Pipeline Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*K25z9pKDPH7lIfk9PWA3Ig.jpeg" /></figure><h3>Step 7: Deploy SonarQube</h3><p>SonarQube provides continuous code quality inspection and security analysis (SAST).</p><pre># Add SonarQube Helm repository<br>helm repo add sonarqube https://SonarSource.github.io/helm-chart-sonarqube<br>helm repo update<br><br># 2. Deploy standalone PostgreSQL first<br>cat &gt; postgres-for-sonarqube.yaml &lt;&lt;&#39;EOF&#39;<br>apiVersion: apps/v1<br>kind: StatefulSet<br>metadata:<br>  name: postgres-sonarqube<br>  namespace: sonarqube<br>spec:<br>  serviceName: postgres-sonarqube<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: postgres-sonarqube<br>  template:<br>    metadata:<br>      labels:<br>        app: postgres-sonarqube<br>    spec:<br>      containers:<br>      - name: postgres<br>        image: postgres:15-alpine<br>        env:<br>        - name: POSTGRES_USER<br>          value: sonarUser<br>        - name: POSTGRES_PASSWORD<br>          value: sonarpass<br>        - name: POSTGRES_DB<br>          value: sonarDB<br>        - name: PGDATA<br>          value: /var/lib/postgresql/data/pgdata<br>        ports:<br>        - containerPort: 5432<br>        volumeMounts:<br>        - name: postgres-storage<br>          mountPath: /var/lib/postgresql/data<br>        resources:<br>          requests:<br>            cpu: 250m<br>            memory: 512Mi<br>          limits:<br>            cpu: 1000m<br>            memory: 1Gi<br>  volumeClaimTemplates:<br>  - metadata:<br>      name: postgres-storage<br>    spec:<br>      accessModes: [ &quot;ReadWriteOnce&quot; ]<br>      storageClassName: gp3<br>      resources:<br>        requests:<br>          storage: 20Gi<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: postgres-sonarqube<br>  namespace: sonarqube<br>spec:<br>  selector:<br>    app: postgres-sonarqube<br>  ports:<br>  - port: 5432<br>    targetPort: 5432<br>  clusterIP: None<br>EOF<br><br>kubectl apply -f postgres-for-sonarqube.yaml<br><br># Wait for PostgreSQL to be ready<br>kubectl wait --for=condition=ready pod -l app=postgres-sonarqube -n sonarqube --timeout=300s<br><br># 3. Install SonarQube WITHOUT bundled PostgreSQL<br>cat &gt; sonarqube-values-standalone.yaml &lt;&lt;&#39;EOF&#39;<br>replicaCount: 1<br># DISABLE bundled PostgreSQL<br>postgresql:<br>  enabled: false<br># Use standalone PostgreSQL<br>jdbcOverwrite:<br>  enable: true<br>  jdbcUrl: &quot;jdbc:postgresql://postgres-sonarqube.sonarqube.svc.cluster.local:5432/sonarDB&quot;<br>  jdbcUsername: &quot;sonarUser&quot;<br>  jdbcPassword: &quot;sonarpass&quot;<br>image:<br>  repository: sonarqube<br>  tag: &quot;10.3.0-community&quot;<br>persistence:<br>  enabled: true<br>  storageClass: gp3<br>  size: 10Gi<br>service:<br>  type: LoadBalancer<br>  port: 9000<br>resources:<br>  requests:<br>    cpu: 500m<br>    memory: 2Gi<br>  limits:<br>    cpu: 2000m<br>    memory: 4Gi<br>startupProbe:<br>  initialDelaySeconds: 90<br>  periodSeconds: 10<br>  failureThreshold: 30<br>livenessProbe:<br>  initialDelaySeconds: 90<br>  periodSeconds: 30<br>readinessProbe:<br>  initialDelaySeconds: 90<br>  periodSeconds: 30<br>  failureThreshold: 6<br>EOF<br>helm upgrade --install sonarqube sonarqube/sonarqube \<br>  --namespace sonarqube \<br>  --values sonarqube-values-standalone.yaml \<br>  --version 10.2.0 \<br>  --wait \<br>  --timeout 15m</pre><h3>Step 8: Access and Configure SonarQube</h3><pre># Get SonarQube URL<br>SONARQUBE_URL=$(kubectl get svc sonarqube-sonarqube -n sonarqube -o jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39;)<br>echo &quot;SonarQube URL: http://${SONARQUBE_URL}:9000&quot;<br># Or use port-forward<br>kubectl port-forward svc/sonarqube-sonarqube -n sonarqube 9000:9000<br># Access at: http://localhost:9000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*madwngTo85GfqngdkpQVLQ.png" /></figure><p><strong>In SonarQube UI (http://localhost:9000):</strong></p><p><strong>First Login</strong></p><ul><li>Username: admin</li><li>Password: admin</li><li>You’ll be prompted to change the password</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wDrWAUPZSsd9hkk4l5M5Ag.png" /></figure><p><strong>Generate Token for GitHub Actions:</strong></p><ul><li>Click on your profile (top right)</li><li>My Account → Security → Generate Token</li><li>Token Name: github-actions</li><li>Type: User Token</li><li>Click Generate</li><li><strong>COPY AND SAVE THIS TOKEN</strong> (you’ll need it for GitHub Secrets)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJ1-O1kEHSPbgWOASAXDFA.png" /></figure><p><strong>Create Backend Project:</strong></p><ul><li>Click “Create Project” → “Manually”</li><li>Project key: tax-calculator-backend</li><li>Display name: Tax Calculator Backend</li><li>Click “Set Up”</li></ul><p><strong>Create Frontend Project:</strong></p><ul><li>Click “Create Project” → “Manually”</li><li>Project key: tax-calculator-frontend</li><li>Display name: Tax Calculator Frontend</li><li>Click “Set Up”</li></ul><h3>Step 9: Prepare GitHub Repository</h3><p>Ensure your repository has this structure:</p><pre>vault-gitops-platform/<br>├── .github/workflows/               # GitHub Actions CI/CD pipelines<br>│   ├── backend-ci.yml<br>│   └── frontend-ci.yml<br>├── argocd/applications/             # ArgoCD Application definitions<br>│   ├── tax-calculator-backend.yaml<br>│   └── tax-calculator-frontend.yaml<br>├── kubernetes/base/                 # Base Kubernetes manifests<br>│   └── tax-calculator/<br>│       ├── backend/<br>│       │   ├── deployment.yaml<br>│       │   └── service.yaml<br>│       └── frontend/<br>│           ├── deployment.yaml<br>│           └── service.yaml<br>├── tax-calculator-app/              # Application source code<br>│   ├── backend/<br>│   │   ├── Dockerfile              # Multi-stage, security-hardened<br>│   │   ├── go.mod<br>│   │   ├── main.go<br>│   │   └── sonar-project.properties<br>│   └── frontend/<br>│       ├── Dockerfile              # Multi-stage, security-hardened<br>│       ├── nginx.conf<br>│       ├── package.json<br>│       └── sonar-project.properties<br>├── helm-values/                     # Helm values for observability<br>│   ├── prometheus-values.yaml<br>│   └── loki-values.yaml<br>├── k8s/                            # Part 1<br>│   ├── All codes to implement Part 1<br>├── terraform/                       # Infrastructure as Code <br>│   ├── environments/dev/<br>│   └── modules/<br>├── scripts/                         # Helper scripts<br>│   └── generate-gosum.sh<br>├── README.md<br>└── QUICK_REFERENCE.md</pre><h3>Step 9: Setup GitHub Secrets</h3><p>In your GitHub repository:</p><ol><li>Go to <strong>Settings → Secrets and variables → Actions → New repository secret</strong></li><li>Add these secrets:</li></ol><pre>Name: AWS_ACCESS_KEY_ID<br>Value: [Your AWS Access Key]<br>Name: AWS_SECRET_ACCESS_KEY<br>Value: [Your AWS Secret Key]<br>Name: AWS_REGION<br>Value: eu-west-2<br>Name: ECR_REGISTRY<br>Value: [Your AWS Account ID].dkr.ecr.eu-west-2.amazonaws.com<br>Name: SONAR_HOST_URL<br>Value: http://[SonarQube LoadBalancer URL]:9000<br>Name: SONAR_TOKEN<br>Value: [Token generated in Step 8]<br>Name: ARGOCD_SERVER<br>Value: [ArgoCD LoadBalancer URL]<br>Name: ARGOCD_AUTH_TOKEN<br>Value: [Generate this in next step]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ngdsR1H112mBK6SH2i-Xg.png" /></figure><h3>Step 10: Generate ArgoCD Token for GitHub Actions</h3><pre># Create service account for GitHub Actions<br>kubectl create serviceaccount github-actions -n argocd<br><br># Create role with necessary permissions<br>cat &gt; argocd-github-actions-role.yaml &lt;&lt;&#39;EOF&#39;<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: github-actions<br>  namespace: argocd<br>rules:<br>  - apiGroups: [&quot;argoproj.io&quot;]<br>    resources: [&quot;applications&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;update&quot;, &quot;sync&quot;, &quot;patch&quot;]<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;secrets&quot;, &quot;configmaps&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;]<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: github-actions<br>  namespace: argocd<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: Role<br>  name: github-actions<br>subjects:<br>  - kind: ServiceAccount<br>    name: github-actions<br>    namespace: argocd<br>EOF<br><br>kubectl apply -f argocd-github-actions-role.yaml<br><br># Generate token for the service account<br>ARGOCD_TOKEN=$(kubectl create token github-actions -n argocd --duration=87600h)<br>echo &quot;ArgoCD Token for GitHub Actions:&quot;<br>echo &quot;${ARGOCD_TOKEN}&quot;<br>echo &quot;&quot;<br>echo &quot;Add this as ARGOCD_AUTH_TOKEN in GitHub Secrets&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Buwnp_9SOqyw8ktiYuSbBg.png" /></figure><h3>Step 11: Create SonarQube Configuration for Backend</h3><p>Create tax-calculator-app/backend/sonar-project.properties:</p><pre># Project identification<br>sonar.projectKey=tax-calculator-backend<br>sonar.projectName=Tax Calculator Backend<br>sonar.projectVersion=1.0<br># Source code<br>sonar.sources=.<br>sonar.exclusions=**/*_test.go,**/vendor/**,**/*.pb.go<br># Tests<br>sonar.tests=.<br>sonar.test.inclusions=**/*_test.go<br># Coverage<br>sonar.go.coverage.reportPaths=coverage.out<br>sonar.coverageReportPaths=coverage.out<br># Language<br>sonar.language=go<br>sonar.sourceEncoding=UTF-8<br># Quality Gates<br>sonar.qualitygate.wait=true<br>sonar.qualitygate.timeout=300</pre><h3>Step 12: Create Backend CI/CD Workflow</h3><p>Create .github/workflows/backend-ci.yml:</p><pre>name: Backend CI/CD<br>on:<br>  push:<br>    branches: [main, develop]<br>    paths:<br>      - &#39;tax-calculator-app/backend/**&#39;<br>      - &#39;.github/workflows/backend-ci.yml&#39;<br>  pull_request:<br>    branches: [main]<br>    paths:<br>      - &#39;tax-calculator-app/backend/**&#39;<br>env:<br>  GO_VERSION: &#39;1.21&#39;<br>  IMAGE_NAME: tax-calculator-backend<br>  AWS_REGION: eu-west-2<br>jobs:<br>  # Job 1: Code Quality Analysis with SonarQube<br>  sonarqube-analysis:<br>    name: SonarQube Code Quality<br>    runs-on: ubuntu-latest<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br>        with:<br>          fetch-depth: 0  # Full history for better analysis<br>      - name: Set up Go<br>        uses: actions/setup-go@v4<br>        with:<br>          go-version: ${{ env.GO_VERSION }}<br>      - name: Cache Go modules<br>        uses: actions/cache@v3<br>        with:<br>          path: ~/go/pkg/mod<br>          key: ${{ runner.os }}-go-${{ hashFiles(&#39;**/go.sum&#39;) }}<br>          restore-keys: |<br>            ${{ runner.os }}-go-<br>      - name: Install dependencies<br>        working-directory: ./tax-calculator-app/backend<br>        run: go mod download<br>      - name: Run tests with coverage<br>        working-directory: ./tax-calculator-app/backend<br>        run: |<br>          go test -v -coverprofile=coverage.out -covermode=atomic ./...<br>          go tool cover -func=coverage.out<br>      - name: SonarQube Scan<br>        uses: SonarSource/sonarqube-scan-action@master<br>        env:<br>          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}<br>        with:<br>          projectBaseDir: ./tax-calculator-app/backend<br>          args: &gt;<br>            -Dsonar.projectKey=tax-calculator-backend<br>            -Dsonar.go.coverage.reportPaths=coverage.out<br>            -Dsonar.sources=.<br>            -Dsonar.exclusions=**/*_test.go,**/vendor/**<br>      - name: Quality Gate Check<br>        uses: SonarSource/sonarqube-quality-gate-action@master<br>        timeout-minutes: 5<br>        env:<br>          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}<br>  # Job 2: Build Docker Image and Security Scan<br>  build-and-scan:<br>    name: Build Image &amp; Security Scan<br>    runs-on: ubuntu-latest<br>    needs: sonarqube-analysis<br>    if: github.event_name == &#39;push&#39;<br>    <br>    outputs:<br>      image-tag: ${{ steps.meta.outputs.version }}<br>      image-digest: ${{ steps.build.outputs.digest }}<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br>      - name: Set up Docker Buildx<br>        uses: docker/setup-buildx-action@v3<br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          aws-region: ${{ env.AWS_REGION }}<br>      - name: Login to Amazon ECR<br>        id: login-ecr<br>        uses: aws-actions/amazon-ecr-login@v2<br>      - name: Extract Docker metadata<br>        id: meta<br>        uses: docker/metadata-action@v5<br>        with:<br>          images: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}<br>          tags: |<br>            type=ref,event=branch<br>            type=sha,prefix={{branch}}-<br>            type=semver,pattern={{version}}<br>            type=raw,value=latest,enable={{is_default_branch}}<br>      - name: Build Docker image<br>        id: build<br>        uses: docker/build-push-action@v5<br>        with:<br>          context: ./tax-calculator-app/backend<br>          file: ./tax-calculator-app/backend/Dockerfile<br>          push: false<br>          load: true<br>          tags: ${{ steps.meta.outputs.tags }}<br>          labels: ${{ steps.meta.outputs.labels }}<br>          cache-from: type=gha<br>          cache-to: type=gha,mode=max<br>          build-args: |<br>            VERSION=${{ steps.meta.outputs.version }}<br>      - name: Run Trivy vulnerability scanner<br>        uses: aquasecurity/trivy-action@master<br>        with:<br>          image-ref: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}<br>          format: &#39;sarif&#39;<br>          output: &#39;trivy-results.sarif&#39;<br>          severity: &#39;CRITICAL,HIGH&#39;<br>          exit-code: &#39;1&#39;  # Fail build on critical/high vulnerabilities<br>      - name: Upload Trivy results to GitHub Security<br>        uses: github/codeql-action/upload-sarif@v2<br>        if: always()<br>        with:<br>          sarif_file: &#39;trivy-results.sarif&#39;<br>      - name: Run Trivy for detailed report<br>        uses: aquasecurity/trivy-action@master<br>        with:<br>          image-ref: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}<br>          format: &#39;table&#39;<br>          severity: &#39;CRITICAL,HIGH,MEDIUM&#39;<br>      - name: Push image to Amazon ECR<br>        if: success()<br>        id: push<br>        uses: docker/build-push-action@v5<br>        with:<br>          context: ./tax-calculator-app/backend<br>          file: ./tax-calculator-app/backend/Dockerfile<br>          push: true<br>          tags: ${{ steps.meta.outputs.tags }}<br>          labels: ${{ steps.meta.outputs.labels }}<br>          cache-from: type=gha<br>          cache-to: type=gha,mode=max<br>  # Job 3: Update Kubernetes Manifests<br>  update-manifests:<br>    name: Update K8s Manifests<br>    runs-on: ubuntu-latest<br>    needs: build-and-scan<br>    if: github.event_name == &#39;push&#39; &amp;&amp; github.ref == &#39;refs/heads/main&#39;<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br>        with:<br>          token: ${{ secrets.GITHUB_TOKEN }}<br>      - name: Update deployment manifest<br>        run: |<br>          IMAGE_TAG=&quot;${{ needs.build-and-scan.outputs.image-tag }}&quot;<br>          echo &quot;Updating backend deployment to use image tag: ${IMAGE_TAG}&quot;<br>          <br>          sed -i &quot;s|image:.*backend.*|image: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}:${IMAGE_TAG}|g&quot; \<br>            kubernetes/base/tax-calculator/backend/deployment.yaml<br>          <br>          cat kubernetes/base/tax-calculator/backend/deployment.yaml<br>      - name: Commit and push changes<br>        run: |<br>          git config --local user.email &quot;github-actions[bot]@users.noreply.github.com&quot;<br>          git config --local user.name &quot;github-actions[bot]&quot;<br>          <br>          git add kubernetes/base/tax-calculator/backend/deployment.yaml<br>          <br>          if git diff --staged --quiet; then<br>            echo &quot;No changes to commit&quot;<br>          else<br>            git commit -m &quot;chore(backend): update image to ${{ needs.build-and-scan.outputs.image-tag }}&quot;<br>            git push<br>          fi<br>  # Job 4: Trigger ArgoCD Sync<br>  argocd-sync:<br>    name: Deploy via ArgoCD<br>    runs-on: ubuntu-latest<br>    needs: update-manifests<br>    <br>    steps:<br>      - name: Install ArgoCD CLI<br>        run: |<br>          curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64<br>          chmod +x argocd<br>          sudo mv argocd /usr/local/bin/<br>          argocd version --client<br>      - name: Sync ArgoCD application<br>        run: |<br>          argocd login ${{ secrets.ARGOCD_SERVER }} \<br>            --auth-token ${{ secrets.ARGOCD_AUTH_TOKEN }} \<br>            --grpc-web \<br>            --insecure<br>          <br>          echo &quot;Syncing tax-calculator-backend application...&quot;<br>          argocd app sync tax-calculator-backend --prune<br>          <br>          echo &quot;Waiting for application to be healthy...&quot;<br>          argocd app wait tax-calculator-backend --timeout 300<br>      - name: Get deployment status<br>        run: |<br>          argocd app get tax-calculator-backend</pre><h3>Step 13: Create ArgoCD Application for Backend</h3><p>Create argocd/applications/tax-calculator-backend.yaml:</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: tax-calculator-backend<br>  namespace: argocd<br>  finalizers:<br>    - resources-finalizer.argocd.argoproj.io<br>spec:<br>  project: default<br>  <br>  source:<br>    repoURL: https://github.com/YOUR_USERNAME/vault-gitops-platform.git  # UPDATE THIS<br>    targetRevision: HEAD<br>    path: kubernetes/base/tax-calculator/backend<br>  <br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: tax-calculator<br>  <br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true<br>      allowEmpty: false<br>    syncOptions:<br>      - CreateNamespace=true<br>      - PrunePropagationPolicy=foreground<br>      - PruneLast=true<br>    retry:<br>      limit: 5<br>      backoff:<br>        duration: 5s<br>        factor: 2<br>        maxDuration: 3m<br>  <br>  # Ignore HPA-controlled replica count<br>  ignoreDifferences:<br>    - group: apps<br>      kind: Deployment<br>      jsonPointers:<br>        - /spec/replicas</pre><p>Apply the ArgoCD application:</p><pre># Apply<br>kubectl apply -f argocd/applications/tax-calculator-backend.yaml<br># Verify<br>argocd app get tax-calculator-backend</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*vKTgzecZnlYh8u8XZzKNrg.png" /></figure><h3>Step 14: Create SonarQube Configuration for Frontend</h3><p>Create tax-calculator-app/frontend/sonar-project.properties:</p><pre># Project identification<br>sonar.projectKey=tax-calculator-frontend<br>sonar.projectName=Tax Calculator Frontend<br>sonar.projectVersion=1.0<br># Source code<br>sonar.sources=src<br>sonar.exclusions=**/*.test.js,**/*.test.jsx,**/node_modules/**,**/build/**,**/dist/**,**/*.config.js<br># Tests<br>sonar.tests=src<br>sonar.test.inclusions=**/*.test.js,**/*.test.jsx<br># Coverage<br>sonar.javascript.lcov.reportPaths=coverage/lcov.info<br># Language<br>sonar.language=js<br>sonar.sourceEncoding=UTF-8<br># Quality Gates<br>sonar.qualitygate.wait=true<br>sonar.qualitygate.timeout=300</pre><h3>Step 15: Create Frontend CI/CD Workflow</h3><p>Create .github/workflows/frontend-ci.yml:</p><pre>name: Frontend CI/CD<br>on:<br>  push:<br>    branches: [main, develop]<br>    paths:<br>      - &#39;tax-calculator-app/frontend/**&#39;<br>      - &#39;.github/workflows/frontend-ci.yml&#39;<br>  pull_request:<br>    branches: [main]<br>    paths:<br>      - &#39;tax-calculator-app/frontend/**&#39;<br>env:<br>  NODE_VERSION: &#39;18&#39;<br>  IMAGE_NAME: tax-calculator-frontend<br>  AWS_REGION: eu-west-2<br>jobs:<br>  # Job 1: Code Quality and Tests<br>  test-and-scan:<br>    name: Test &amp; SonarQube Scan<br>    runs-on: ubuntu-latest<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br>        with:<br>          fetch-depth: 0<br>      - name: Setup Node.js<br>        uses: actions/setup-node@v4<br>        with:<br>          node-version: ${{ env.NODE_VERSION }}<br>          cache: &#39;npm&#39;<br>          cache-dependency-path: &#39;./tax-calculator-app/frontend/package-lock.json&#39;<br>      - name: Install dependencies<br>        working-directory: ./tax-calculator-app/frontend<br>        run: npm ci<br>      - name: Run linting<br>        working-directory: ./tax-calculator-app/frontend<br>        run: npm run lint || true<br>      - name: Run tests with coverage<br>        working-directory: ./tax-calculator-app/frontend<br>        run: npm test -- --coverage --watchAll=false --passWithNoTests<br>      - name: SonarQube Scan<br>        uses: SonarSource/sonarqube-scan-action@master<br>        env:<br>          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}<br>        with:<br>          projectBaseDir: ./tax-calculator-app/frontend<br>          args: &gt;<br>            -Dsonar.projectKey=tax-calculator-frontend<br>            -Dsonar.javascript.lcov.reportPaths=coverage/lcov.info<br>            -Dsonar.sources=src<br>            -Dsonar.exclusions=**/*.test.js,**/*.test.jsx,**/node_modules/**<br>      - name: Quality Gate Check<br>        uses: SonarSource/sonarqube-quality-gate-action@master<br>        timeout-minutes: 5<br>        env:<br>          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}<br>  # Job 2: Build and Security Scan<br>  build-and-scan:<br>    name: Build Image &amp; Security Scan<br>    runs-on: ubuntu-latest<br>    needs: test-and-scan<br>    if: github.event_name == &#39;push&#39;<br>    <br>    outputs:<br>      image-tag: ${{ steps.meta.outputs.version }}<br>      image-digest: ${{ steps.build.outputs.digest }}<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br>      - name: Set up Docker Buildx<br>        uses: docker/setup-buildx-action@v3<br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>          aws-region: ${{ env.AWS_REGION }}<br>      - name: Login to Amazon ECR<br>        id: login-ecr<br>        uses: aws-actions/amazon-ecr-login@v2<br>      - name: Extract Docker metadata<br>        id: meta<br>        uses: docker/metadata-action@v5<br>        with:<br>          images: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}<br>          tags: |<br>            type=ref,event=branch<br>            type=sha,prefix={{branch}}-<br>            type=semver,pattern={{version}}<br>            type=raw,value=latest,enable={{is_default_branch}}<br>      - name: Build Docker image<br>        id: build<br>        uses: docker/build-push-action@v5<br>        with:<br>          context: ./tax-calculator-app/frontend<br>          file: ./tax-calculator-app/frontend/Dockerfile<br>          push: false<br>          load: true<br>          tags: ${{ steps.meta.outputs.tags }}<br>          labels: ${{ steps.meta.outputs.labels }}<br>          cache-from: type=gha<br>          cache-to: type=gha,mode=max<br>      - name: Run Trivy vulnerability scanner<br>        uses: aquasecurity/trivy-action@master<br>        with:<br>          image-ref: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}<br>          format: &#39;sarif&#39;<br>          output: &#39;trivy-results.sarif&#39;<br>          severity: &#39;CRITICAL,HIGH&#39;<br>          exit-code: &#39;1&#39;<br>      - name: Upload Trivy results to GitHub Security<br>        uses: github/codeql-action/upload-sarif@v2<br>        if: always()<br>        with:<br>          sarif_file: &#39;trivy-results.sarif&#39;<br>      - name: Run Trivy for detailed report<br>        uses: aquasecurity/trivy-action@master<br>        with:<br>          image-ref: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}<br>          format: &#39;table&#39;<br>          severity: &#39;CRITICAL,HIGH,MEDIUM&#39;<br>      - name: Push image to Amazon ECR<br>        if: success()<br>        id: push<br>        uses: docker/build-push-action@v5<br>        with:<br>          context: ./tax-calculator-app/frontend<br>          file: ./tax-calculator-app/frontend/Dockerfile<br>          push: true<br>          tags: ${{ steps.meta.outputs.tags }}<br>          labels: ${{ steps.meta.outputs.labels }}<br>          cache-from: type=gha<br>          cache-to: type=gha,mode=max<br>  # Job 3: Update Kubernetes Manifests<br>  update-manifests:<br>    name: Update K8s Manifests<br>    runs-on: ubuntu-latest<br>    needs: build-and-scan<br>    if: github.event_name == &#39;push&#39; &amp;&amp; github.ref == &#39;refs/heads/main&#39;<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br>        with:<br>          token: ${{ secrets.GITHUB_TOKEN }}<br>      - name: Update deployment manifest<br>        run: |<br>          IMAGE_TAG=&quot;${{ needs.build-and-scan.outputs.image-tag }}&quot;<br>          echo &quot;Updating frontend deployment to use image tag: ${IMAGE_TAG}&quot;<br>          <br>          sed -i &quot;s|image:.*frontend.*|image: ${{ secrets.ECR_REGISTRY }}/${{ env.IMAGE_NAME }}:${IMAGE_TAG}|g&quot; \<br>            kubernetes/base/tax-calculator/frontend/deployment.yaml<br>          <br>          cat kubernetes/base/tax-calculator/frontend/deployment.yaml<br>      - name: Commit and push changes<br>        run: |<br>          git config --local user.email &quot;github-actions[bot]@users.noreply.github.com&quot;<br>          git config --local user.name &quot;github-actions[bot]&quot;<br>          <br>          git add kubernetes/base/tax-calculator/frontend/deployment.yaml<br>          <br>          if git diff --staged --quiet; then<br>            echo &quot;No changes to commit&quot;<br>          else<br>            git commit -m &quot;chore(frontend): update image to ${{ needs.build-and-scan.outputs.image-tag }}&quot;<br>            git push<br>          fi<br>  # Job 4: Trigger ArgoCD Sync<br>  argocd-sync:<br>    name: Deploy via ArgoCD<br>    runs-on: ubuntu-latest<br>    needs: update-manifests<br>    <br>    steps:<br>      - name: Install ArgoCD CLI<br>        run: |<br>          curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64<br>          chmod +x argocd<br>          sudo mv argocd /usr/local/bin/<br>          argocd version --client<br>      - name: Sync ArgoCD application<br>        run: |<br>          argocd login ${{ secrets.ARGOCD_SERVER }} \<br>            --auth-token ${{ secrets.ARGOCD_AUTH_TOKEN }} \<br>            --grpc-web \<br>            --insecure<br>          <br>          echo &quot;Syncing tax-calculator-frontend application...&quot;<br>          argocd app sync tax-calculator-frontend --prune<br>          <br>          echo &quot;Waiting for application to be healthy...&quot;<br>          argocd app wait tax-calculator-frontend --timeout 300<br>      - name: Get deployment status<br>        run: |<br>          argocd app get tax-calculator-frontend</pre><h3>Step 16: Create ArgoCD Application for Frontend</h3><p>Create argocd/applications/tax-calculator-frontend.yaml:</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: tax-calculator-frontend<br>  namespace: argocd<br>  finalizers:<br>    - resources-finalizer.argocd.argoproj.io<br>spec:<br>  project: default<br>  <br>  source:<br>    repoURL: https://github.com/YOUR_USERNAME/vault-gitops-platform.git  # UPDATE THIS<br>    targetRevision: HEAD<br>    path: kubernetes/base/tax-calculator/frontend<br>  <br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: tax-calculator<br>  <br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true<br>      allowEmpty: false<br>    syncOptions:<br>      - CreateNamespace=true<br>      - PrunePropagationPolicy=foreground<br>      - PruneLast=true<br>    retry:<br>      limit: 5<br>      backoff:<br>        duration: 5s<br>        factor: 2<br>        maxDuration: 3m<br>  <br>  # Ignore HPA-controlled replica count<br>  ignoreDifferences:<br>    - group: apps<br>      kind: Deployment<br>      jsonPointers:<br>        - /spec/replicas</pre><p>Apply the ArgoCD application:</p><pre># Apply<br>kubectl apply -f argocd/applications/tax-calculator-frontend.yaml<br># Verify<br>argocd app get tax-calculator-frontend</pre><h3>Step 17: Create Nginx Configuration for Frontend</h3><p>Create tax-calculator-app/frontend/nginx.conf:</p><pre>user nginx;<br>worker_processes auto;<br>error_log /var/log/nginx/error.log warn;<br>pid /var/run/nginx.pid;<br>events {<br>    worker_connections 1024;<br>}<br>http {<br>    include /etc/nginx/mime.types;<br>    default_type application/octet-stream;<br>    log_format main &#39;$remote_addr - $remote_user [$time_local] &quot;$request&quot; &#39;<br>                    &#39;$status $body_bytes_sent &quot;$http_referer&quot; &#39;<br>                    &#39;&quot;$http_user_agent&quot; &quot;$http_x_forwarded_for&quot;&#39;;<br>    access_log /var/log/nginx/access.log main;<br>    sendfile on;<br>    tcp_nopush on;<br>    keepalive_timeout 65;<br>    gzip on;<br>    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;<br>    server {<br>        listen 8080;<br>        server_name _;<br>        root /usr/share/nginx/html;<br>        index index.html;<br>        # Security headers<br>        add_header X-Frame-Options &quot;SAMEORIGIN&quot; always;<br>        add_header X-Content-Type-Options &quot;nosniff&quot; always;<br>        add_header X-XSS-Protection &quot;1; mode=block&quot; always;<br>        add_header Referrer-Policy &quot;no-referrer-when-downgrade&quot; always;<br>        # SPA routing<br>        location / {<br>            try_files $uri $uri/ /index.html;<br>        }<br>        # API proxy (if needed)<br>        location /api {<br>            proxy_pass http://backend:8080;<br>            proxy_http_version 1.1;<br>            proxy_set_header Upgrade $http_upgrade;<br>            proxy_set_header Connection &#39;upgrade&#39;;<br>            proxy_set_header Host $host;<br>            proxy_cache_bypass $http_upgrade;<br>        }<br>        # Health check endpoint<br>        location /health {<br>            access_log off;<br>            return 200 &quot;healthy\n&quot;;<br>            add_header Content-Type text/plain;<br>        }<br>    }<br>}</pre><h3>Step 18: Test the Complete CI/CD Pipeline (Both Backend and Frontend)</h3><pre># Make a test change to trigger the pipeline<br>cd tax-calculator-app/backend<br>echo &quot;// CI/CD Pipeline Test - $(date)&quot; &gt;&gt; main.go<br># Commit and push<br>git add .<br>git commit -m &quot;test: trigger CI/CD pipeline&quot;<br>git push origin main<br>echo &quot;Pipeline triggered! Watch progress at:&quot;<br>echo &quot;1. GitHub Actions: https://github.com/YOUR_USERNAME/vault-gitops-platform/actions&quot;<br>echo &quot;2. SonarQube: http://localhost:9000&quot;<br>echo &quot;3. ArgoCD: https://${ARGOCD_SERVER}&quot;</pre><p><strong>Monitor the Pipeline:</strong></p><pre># Watch GitHub Actions<br>gh run watch<br><br># Watch ArgoCD sync<br>argocd app get tax-calculator-backend --refresh<br>argocd app wait tax-calculator-backend<br># Watch pods rolling out<br>kubectl get pods -n tax-calculator -w</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iW105Z0rfYKJfK1hFN9JKg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bH0u9QajDmzeFAoxmmhySw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X78ZuQpyu83n0f3q0h1qzw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0NOe7R4Wwobq2Ipb5ONSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R2zX0aFLSLYsrEAtKGgz4g.png" /></figure><h3>Part 4: Observability Stack</h3><blockquote><strong>Metrics + Logs = Visibility</strong><br>Production systems need more than “it works” — they need to answer:</blockquote><blockquote><strong>How is it performing?</strong> (Metrics → Prometheus)</blockquote><blockquote><strong>What happened?</strong> (Logs → Loki)</blockquote><blockquote><strong>Is something wrong?</strong> (Alerts → AlertManager)</blockquote><blockquote><strong>Unified Stack</strong>: Prometheus for metrics, Loki for logs — both use similar query languages and understand Kubernetes natively. Grafana visualises everything. This isn’t just monitoring; it’s designing systems to be debuggable.</blockquote><h3>Step 19: Deploy Prometheus Stack</h3><p>Prometheus is the de facto standard for Kubernetes monitoring.</p><pre># Add Prometheus Helm repository<br>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update<br># Create Prometheus values file<br>cat &gt; prometheus-values.yaml &lt;&lt;&#39;EOF&#39;<br># Prometheus configuration<br>prometheus:<br>  prometheusSpec:<br>    retention: 30d<br>    retentionSize: &quot;50GB&quot;<br>    <br>    # Persistent storage<br>    storageSpec:<br>      volumeClaimTemplate:<br>        spec:<br>          accessModes: [&quot;ReadWriteOnce&quot;]<br>          storageClassName: gp3<br>          resources:<br>            requests:<br>              storage: 50Gi<br>    <br>    # Resource limits<br>    resources:<br>      requests:<br>        cpu: 500m<br>        memory: 2Gi<br>      limits:<br>        cpu: 2000m<br>        memory: 4Gi<br>    <br>    # Scrape all ServiceMonitors/PodMonitors<br>    serviceMonitorSelectorNilUsesHelmValues: false<br>    podMonitorSelectorNilUsesHelmValues: false<br>    <br>    # Additional scrape configs<br>    additionalScrapeConfigs:<br>      - job_name: &#39;kubernetes-pods&#39;<br>        kubernetes_sd_configs:<br>          - role: pod<br>        relabel_configs:<br>          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]<br>            action: keep<br>            regex: true<br># Grafana configuration<br>grafana:<br>  enabled: true<br>  adminPassword: &quot;admin123&quot;  # CHANGE THIS IN PRODUCTION!<br>  <br>  persistence:<br>    enabled: true<br>    storageClassName: gp3<br>    size: 10Gi<br>  <br>  service:<br>    type: LoadBalancer<br>    port: 80<br>  <br>  # Pre-install dashboards<br>  defaultDashboardsEnabled: true<br>  <br>  # Dashboard sidecar<br>  sidecar:<br>    dashboards:<br>      enabled: true<br>      label: grafana_dashboard<br>      searchNamespace: ALL<br>    datasources:<br>      enabled: true<br>      label: grafana_datasource<br># AlertManager configuration<br>alertmanager:<br>  enabled: true<br>  <br>  alertmanagerSpec:<br>    storage:<br>      volumeClaimTemplate:<br>        spec:<br>          accessModes: [&quot;ReadWriteOnce&quot;]<br>          storageClassName: gp3<br>          resources:<br>            requests:<br>              storage: 10Gi<br>    <br>    resources:<br>      requests:<br>        cpu: 100m<br>        memory: 128Mi<br>      limits:<br>        cpu: 200m<br>        memory: 256Mi<br># Additional components<br>kubeStateMetrics:<br>  enabled: true<br>nodeExporter:<br>  enabled: true<br>EOF<br># Install Prometheus stack (this takes 5-10 minutes)<br>echo &quot;Installing Prometheus stack...&quot;<br>helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \<br>  --namespace monitoring \<br>  --values prometheus-values.yaml \<br>  --version 55.5.0 \<br>  --wait \<br>  --timeout 10m<br># Verify deployment<br>kubectl get pods -n monitoring</pre><h3>Step 20: Access Grafana</h3><pre># Get Grafana URL<br>GRAFANA_URL=$(kubectl get svc kube-prometheus-stack-grafana -n monitoring -o jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39;)<br>echo &quot;Grafana URL: http://${GRAFANA_URL}&quot;<br># Or use port-forward<br>kubectl port-forward svc/kube-prometheus-stack-grafana -n monitoring 3000:80<br># Access at: http://localhost:3000<br># Username: admin<br># Password: admin123 (change this!)</pre><p><strong>In Grafana:</strong></p><p><strong>Change Default Password:</strong></p><ul><li>Profile → Change Password</li></ul><p><strong>Explore Pre-installed Dashboards:</strong></p><ul><li>Dashboards → Browse</li></ul><p>Look for:</p><ul><li>Kubernetes / Compute Resources / Cluster</li><li>Kubernetes / Compute Resources / Namespace</li><li>Kubernetes / Compute Resources / Node</li><li>Node Exporter / Nodes</li></ul><p><strong>Verify Data Sources:</strong></p><ul><li>Configuration → Data Sources</li><li>Should see Prometheus (default)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XF1SakaU0hnXgjE6A1KUGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMWl-KuoYdh97YR8lCgZtA.png" /></figure><h3>Step 21: Access Prometheus</h3><pre># Port-forward Prometheus<br>kubectl port-forward svc/kube-prometheus-stack-prometheus -n monitoring 9090:9090<br># Access at: http://localhost:9090</pre><p><strong>Test Queries in Prometheus:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tLHMEml_EMA-KY9o4asyw.png" /></figure><pre># Check all targets are up<br>up<br><br># CPU usage by node<br>node_cpu_seconds_total<br># Pod status<br>kube_pod_status_phase<br># Container memory usage<br>container_memory_usage_bytes<br># API server requests<br>apiserver_request_total</pre><h3>Step 22: Deploy Loki for Log Aggregation</h3><p>Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus.</p><pre># Add Grafana Helm repository<br>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update<br># Create Loki values file<br>cat &gt; loki-values.yaml &lt;&lt;&#39;EOF&#39;<br>loki:<br>  auth_enabled: false<br>  <br>  commonConfig:<br>    replication_factor: 1<br>  <br>  storage:<br>    type: filesystem<br>  <br>  schemaConfig:<br>    configs:<br>      - from: 2024-01-01<br>        store: tsdb<br>        object_store: filesystem<br>        schema: v13<br>        index:<br>          prefix: loki_index_<br>          period: 24h<br># Single binary mode for simplicity<br>singleBinary:<br>  replicas: 1<br>  <br>  persistence:<br>    enabled: true<br>    storageClass: gp3<br>    size: 30Gi<br>  <br>  resources:<br>    requests:<br>      cpu: 500m<br>      memory: 1Gi<br>    limits:<br>      cpu: 1000m<br>      memory: 2Gi<br># Disable gateway<br>gateway:<br>  enabled: false<br># Enable ServiceMonitor for Prometheus<br>monitoring:<br>  serviceMonitor:<br>    enabled: true<br>    labels:<br>      release: kube-prometheus-stack<br># Disable test pod<br>test:<br>  enabled: false<br>EOF<br># Install Loki<br>echo &quot;Installing Loki...&quot;<br>helm upgrade --install loki grafana/loki \<br>  --namespace logging \<br>  --values loki-values.yaml \<br>  --version 5.41.0 \<br>  --wait<br># Install Promtail (log collector)<br>cat &gt; promtail-values.yaml &lt;&lt;&#39;EOF&#39;<br>config:<br>  clients:<br>    - url: http://loki-gateway.logging.svc.cluster.local/loki/api/v1/push<br>daemonset:<br>  enabled: true<br>resources:<br>  requests:<br>    cpu: 100m<br>    memory: 128Mi<br>  limits:<br>    cpu: 200m<br>    memory: 256Mi<br># ServiceMonitor for Prometheus<br>serviceMonitor:<br>  enabled: true<br>  labels:<br>    release: kube-prometheus-stack<br>EOF<br>echo &quot;Installing Promtail...&quot;<br>helm upgrade --install promtail grafana/promtail \<br>  --namespace logging \<br>  --values promtail-values.yaml \<br>  --version 6.15.3 \<br>  --wait<br># Verify deployment<br>kubectl get pods -n logging</pre><h3>Step 23: Configure Loki in Grafana</h3><p>The Loki data source should be automatically configured. Verify:</p><ol><li><strong>In Grafana:</strong> Configuration → Data Sources</li><li><strong>Look for:</strong> Loki</li><li><strong>If not present, add manually:</strong></li></ol><ul><li>Name: Loki</li><li>Type: Loki</li><li>URL: <a href="http://loki-gateway.logging.svc.cluster.local/">http://loki-gateway.logging.svc.cluster.local</a></li></ul><h3>Step 24: Test Log Collection</h3><pre># Generate some logs<br>kubectl run test-logger --image=busybox --restart=Never -- sh -c &quot;while true; do echo &#39;Test log entry at $(date)&#39;; sleep 1; done&quot;<br># Wait a minute, then check in Grafana</pre><p><strong>In Grafana:</strong></p><ol><li>Go to <strong>Explore</strong> (compass icon)</li><li>Select <strong>Loki</strong> data source</li><li>Use query: {namespace=&quot;tax-calculator&quot;}</li><li>Click <strong>Run query</strong></li><li>You should see logs from backend, frontend, and postgres pods</li></ol><p><strong>Test Queries:</strong></p><pre># All logs from tax-calculator namespace<br>{namespace=&quot;tax-calculator&quot;}<br># Backend logs only<br>{namespace=&quot;tax-calculator&quot;, pod=~&quot;backend-.*&quot;}<br># Error logs<br>{namespace=&quot;tax-calculator&quot;} |= &quot;error&quot;<br># JSON logs parsed<br>{namespace=&quot;tax-calculator&quot;} | json<br># Rate of logs<br>rate({namespace=&quot;tax-calculator&quot;}[5m])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ldtUG_oxlr3GeaoPhc11lg.png" /></figure><p><strong>Validation:</strong></p><ul><li>✅ Prometheus collecting metrics from all nodes</li><li>✅ Grafana dashboards showing data</li><li>✅ Loki aggregating logs from all pods</li><li>✅ Can query logs by namespace/pod</li><li>✅ AlertManager ready for alert rules</li></ul><h3>Conclusion</h3><p>You now have a complete, production-grade Kubernetes platform with:</p><p>✅ <strong>End-to-end automation</strong> — 13 minutes commit to production</p><p>✅ <strong>Code quality gates</strong> — 80%+ coverage enforced</p><p>✅ <strong>Security scanning</strong> — Zero critical vulnerabilities</p><p>✅ <strong>Complete observability</strong> — Metrics, logs, dashboards</p><p><strong>GitHub:</strong> <a href="https://github.com/tobiakinlade/vault-gitops-platform.git">https://github.com/tobiakinlade/vault-gitops-platform.git</a></p><p><strong>In Part 3, we will implement:</strong></p><p>✅<strong>Security hardening</strong> — Zero-trust networking</p><p>✅ <strong>Operational excellence</strong> — Backup, DR, autoscaling</p><p>✅ <strong>SRE practices</strong> — SLI/SLO tracking</p><p><strong>Connect with me</strong> on <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">LinkedIn</a> for more AWS, DevOps, and cloud architecture content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df71d3768911" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Production-Grade Application with HashiCorp Vault on AWS EKS]]></title>
            <link>https://medium.com/@tobi.akinlade.co/building-a-production-grade-application-with-hashicorp-vault-on-aws-eks-c9110af7c5fa?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9110af7c5fa</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Tue, 23 Dec 2025 10:41:08 GMT</pubDate>
            <atom:updated>2025-12-23T10:41:08.761Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 1: From Infrastructure to Deployed Application</h3><h3>A Complete Tutorial for Cloud-Native Development with Enterprise Security</h3><h3>What You’ll Build</h3><p>By the end of this tutorial, you’ll have deployed a <strong>production-ready UK tax calculator</strong> that demonstrates:</p><ul><li><strong>Dynamic secrets management</strong> with HashiCorp Vault</li><li><strong>Transit encryption</strong> for sensitive data (PII)</li><li><strong>Kubernetes-native architecture</strong> on AWS EKS</li><li><strong>Infrastructure as Code</strong> with Terraform</li><li><strong>Zero-trust security model</strong> with service account authentication</li><li><strong>High availability</strong> across multiple availability zones</li></ul><p><strong>Live Demo:</strong> A beautiful React frontend with real-time tax calculations, showing encrypted National Insurance numbers and calculation history — all secured by Vault.</p><p><strong>Time Required:</strong> 1–2 hours for complete deployment</p><h3>Architecture Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vvr0X6kPZm2Ie1QvnRJAaQ.png" /></figure><p>Our application consists of three main layers:</p><h3>1. Infrastructure Layer (AWS)</h3><ul><li><strong>EKS Cluster</strong>: Managed Kubernetes for container orchestration</li><li><strong>VPC</strong>: Isolated network with public and private subnets across 3 AZs</li><li><strong>Network Load Balancer</strong>: Internet-facing entry point</li><li><strong>EBS CSI Driver</strong>: Dynamic volume provisioning with gp3 storage</li></ul><h3>2. Security Layer (HashiCorp Vault)</h3><ul><li><strong>Dynamic Database Credentials</strong>: Ephemeral PostgreSQL credentials with 1-hour TTL</li><li><strong>Transit Encryption Engine</strong>: Encrypt National Insurance numbers before storage</li><li><strong>Kubernetes Authentication</strong>: Pod-level identity and access control</li><li><strong>AWS KMS Auto-Unseal</strong>: Automated Vault unsealing using AWS KMS</li></ul><h3>3. Application Layer</h3><ul><li><strong>Go Backend</strong>: REST API with Vault SDK integration</li><li><strong>React Frontend</strong>: Modern UI with Nginx reverse proxy</li><li><strong>PostgreSQL</strong>: Relational database for calculation storage</li></ul><h3>🤔 Why This Architecture?</h3><h3>Why HashiCorp Vault Over Traditional Approaches?</h3><p><strong>Traditional Approach:</strong></p><pre>// ❌ Hardcoded credentials<br>db_password := &quot;MySecretPassword123&quot;</pre><p><strong>Problems:</strong></p><ul><li>Credentials in code/config files</li><li>Manual rotation process</li><li>No audit trail</li><li>Shared credentials across environments</li><li>Long-lived secrets</li></ul><p><strong>Vault Approach:</strong></p><pre>// ✅ Dynamic credentials<br>creds := vault.GetDatabaseCredentials()<br>// Automatically rotated, audited, and ephemeral</pre><p><strong>Benefits:</strong></p><ul><li><strong>Dynamic Secrets</strong>: New credentials for each request, auto-revoked after TTL</li><li><strong>Encryption as a Service</strong>: Never store PII in plaintext</li><li><strong>Centralized Access Control</strong>: Fine-grained policies per service</li><li><strong>Complete Audit Trail</strong>: Who accessed what, when, and why</li><li><strong>Zero-Trust Model</strong>: No long-lived credentials</li></ul><h3>Security Impact Quantified</h3><p><strong>Without Vault (Traditional):</strong></p><pre>🔴 Static credentials: Unlimited lifetime<br>🔴 Rotation: Manual, quarterly (if at all)<br>🔴 Audit: Limited or none<br>🔴 Blast radius: Compromised cred = full DB access forever<br>🔴 PII storage: Plaintext in database</pre><p><strong>With Vault (This Tutorial):</strong></p><pre>🟢 Dynamic credentials: 1-hour auto-expiry<br>🟢 Rotation: Automatic, hourly<br>🟢 Audit: Complete trail of all access<br>🟢 Blast radius: Compromised cred = 1 hour max exposure<br>🟢 PII storage: Encrypted with separate key service</pre><p><strong>Impact:</strong></p><ul><li>8,760 credential rotations per year (vs. 4 manual rotations)</li><li>99.9% reduction in credential exposure time</li><li>100% audit coverage</li><li>Zero plaintext PII in database</li></ul><h3>Prerequisites</h3><h3>Required Tools</h3><pre># Verify installations<br>aws --version        # AWS CLI v2<br>terraform --version  # Terraform v1.0+<br>kubectl version      # kubectl v1.30+<br>docker --version     # Docker 20.10+</pre><h3>AWS Requirements</h3><ul><li>AWS account with appropriate permissions</li><li>AWS CLI configured with credentials</li></ul><h3>Knowledge Level</h3><ul><li><strong>Basic</strong>: Understanding of containers and Kubernetes concepts</li><li><strong>Intermediate</strong>: Familiarity with AWS services</li><li><strong>Advanced</strong>: (Optional) Secrets management concepts</li></ul><h3>Project Structure</h3><p><strong>GitHub:</strong> <a href="https://github.com/tobiakinlade/vault-gitops-platform.git">https://github.com/tobiakinlade/vault-gitops-platform.git</a></p><pre>vault-gitops-platform/<br>├── terraform/<br>│   ├── modules/<br>│   │   ├── vpc/              # Network infrastructure<br>│   │   ├── eks/              # Kubernetes cluster<br>│   │   ├── vault/            # Vault HA deployment<br>│   │   ├── ebs-csi-driver/   # Storage provisioner<br>│   │   └── kms/              # Encryption keys<br>│   └── environments/<br>│       └── dev/              # Development environment<br>├── tax-calculator-app/<br>│   ├── backend/              # Go API<br>│   ├── frontend/             # React UI<br>│   └── k8s/                  # Kubernetes manifests<br>└── docs/                     # Documentation</pre><h3>Run the application locally</h3><pre>Change directory into the application code:<br>cd tax-calculator-app<br><br>Build and start the containers:<br>docker compose up --build</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USr6OiP3IUCwEzd2ADu5_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GLM0Oy1-xXH6Kr-JEAhuyg.png" /></figure><h3>Phase 1: Infrastructure Provisioning</h3><h3>Step 1: Configure Terraform Variables</h3><p>Create terraform/environments/dev/terraform.tfvars:</p><pre># Project Configuration<br>project_name = &quot;tax-calculator&quot;<br>environment  = &quot;dev&quot;<br>aws_region   = &quot;eu-west-2&quot;  # London region<br># VPC Configuration<br>vpc_cidr           = &quot;10.0.0.0/16&quot;<br>availability_zones = [&quot;eu-west-2a&quot;, &quot;eu-west-2b&quot;, &quot;eu-west-2c&quot;]<br># EKS Configuration<br>eks_cluster_version = &quot;1.30&quot;<br># Node Configuration<br>node_instance_types = [&quot;t3.large&quot;]  # 8GB RAM per node<br>node_desired_size   = 3             # 3 nodes for HA<br>node_min_size       = 2             # Minimum for resilience<br>node_max_size       = 4             # Allow autoscaling<br>node_disk_size      = 30            # 30GB per node<br># Vault Configuration<br>vault_namespace     = &quot;vault&quot;<br>vault_replicas      = 2             # HA setup<br>vault_storage_size  = &quot;10Gi&quot;<br>vault_chart_version = &quot;0.28.0&quot;</pre><p><strong>Why These Values?</strong></p><ul><li><strong>t3.large (8GB RAM)</strong>: Sufficient for Vault, app, and system pods without over-provisioning and also observability stack that will be implemented in Part 2 of this series.</li><li><strong>3 nodes</strong>: Allows pod distribution across AZs for true high availability</li><li><strong>2 Vault replicas</strong>: Minimum for Raft consensus (1 leader, 1 follower)</li><li><strong>gp3 storage</strong>: 20% cheaper than gp2, better IOPS performance</li></ul><h3>Step 2: Deploy Infrastructure</h3><pre>cd terraform/environments/dev<br><br># Initialize Terraform<br>terraform init<br># Review the plan (optional but recommended)<br>terraform plan<br># Deploy (~20-25 minutes)<br>terraform apply -auto-approve<br><br># About 63 resources will be provisioned</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*HdxeiyIA_T0tw0Dn3lakKw.png" /></figure><p><strong>What’s Being Created:</strong></p><pre>✅ VPC with public/private subnets (3 AZs)<br>✅ Internet Gateway and NAT Gateways (one per AZ)<br>✅ EKS cluster v1.30 with managed node group<br>✅ EBS CSI driver for dynamic storage<br>✅ KMS keys for encryption (EKS + Vault)<br>✅ Vault HA with Raft storage backend<br>✅ IAM roles with IRSA (service accounts)<br>✅ Security groups and network ACLs<br>✅ CloudWatch log groups</pre><h3>Step 3: Configure kubectl</h3><pre>aws eks update-kubeconfig \<br>  --name tax-calculator-dev-cluster \<br>  --region eu-west-2</pre><h3>Step 4: Verify Infrastructure</h3><pre># Check nodes <br>kubectl get nodes<br><br># Output:<br># NAME                                       STATUS   ROLES    AGE<br># ip-10-0-1-x.eu-west-2.compute.internal     Ready    &lt;none&gt;   5m<br># ip-10-0-2-x.eu-west-2.compute.internal     Ready    &lt;none&gt;   5m<br># ip-10-0-3-x.eu-west-2.compute.internal     Ready    &lt;none&gt;   5m<br># Check EBS CSI driver<br>kubectl get pods -n kube-system | grep ebs<br># Expected: ebs-csi-controller and ebs-csi-node pods Running<br># Check Vault pods<br>kubectl get pods -n vault<br># Expected: vault-0, vault-1 Running<br># Check storage<br>kubectl get storageclass<br># Expected: gp3 (default)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*uN9W_PfP1zK4cqJDeYfWtg.png" /></figure><h3>🔐 Phase 2: HashiCorp Vault Setup</h3><h3>Understanding Vault’s Role</h3><p>Vault sits at the heart of our security model:</p><pre>Application Pod<br>    ↓ (Kubernetes Service Account Token)<br>Vault Kubernetes Auth<br>    ↓ (Validates &amp; Returns Vault Token)<br>Application gets:<br>    - Dynamic DB credentials (1hr TTL)<br>    - Transit encryption key access<br>    - Configuration secrets</pre><h3>Step 1: Initialize Vault</h3><pre>kubectl exec -it vault-0 -n vault -- vault operator init | tee vault-keys.txt</pre><p><strong>Output:</strong></p><pre>Recovery Key 1: abc123def456...<br>Recovery Key 2: ghi789jkl012...<br>Recovery Key 3: mno345pqr678...<br>Recovery Key 4: stu901vwx234...<br>Recovery Key 5: yza567bcd890...<br><br>Initial Root Token: hvs.CAESIJ8xyz...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*4RyWiJA8zqI0ApQsgQ1X8A.png" /></figure><p><strong>CRITICAL: Save vault-keys.txt securely!</strong></p><p><strong>Why Recovery Keys, Not Unseal Keys?</strong></p><p>With AWS KMS auto-unseal:</p><ul><li>Vault unseals automatically using KMS (no manual unseal needed)</li><li>Recovery keys are for disaster recovery only</li><li>Vault pods restart without human intervention</li></ul><h3>Step 2: Generate Root Token</h3><p>Since initial root tokens can be lost, generate a new one:</p><pre># Start generation<br>kubectl exec -it vault-0 -n vault -- vault operator generate-root -init<br># Save the Nonce and OTP<br># Provide 3 recovery keys (run this command 3 times)<br>kubectl exec -it vault-0 -n vault -- vault operator generate-root -nonce=&lt;NONCE&gt;<br># Paste one recovery key each time<br># Decode the encoded token (you get this after 3rd key)<br>kubectl exec -it vault-0 -n vault -- vault operator generate-root \<br>  -decode=&lt;ENCODED_TOKEN&gt; \<br>  -otp=&lt;OTP&gt;<br># Output: Your new root token</pre><h3>Step 3: Login to Vault</h3><pre>export VAULT_TOKEN=&quot;hvs.YOUR_ROOT_TOKEN&quot;<br>kubectl exec -it vault-0 -n vault -- vault login $VAULT_TOKEN</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*59H_JowOAOXTol3kxU5agg.png" /></figure><h3>Step 4: Enable Secret Engines</h3><pre># Database secrets for dynamic credentials<br>kubectl exec -it vault-0 -n vault -- vault secrets enable database<br># Transit encryption for PII<br>kubectl exec -it vault-0 -n vault -- vault secrets enable transit<br># KV store for configuration<br>kubectl exec -it vault-0 -n vault -- vault secrets enable -path=secret kv-v2<br># Kubernetes authentication<br>kubectl exec -it vault-0 -n vault -- vault auth enable kubernetes</pre><p><strong>What Each Engine Does:</strong></p><p>Engine Purpose Example Use <strong>database</strong> Generate DB credentials vault read database/creds/role <strong>transit</strong> Encrypt/decrypt data vault write transit/encrypt/key plaintext=... <strong>kv-v2</strong> Store configuration vault kv put secret/config api_key=xyz <strong>kubernetes</strong> Pod authentication Pods use ServiceAccount tokens</p><h3>Step 5: Configure Kubernetes Authentication</h3><pre>kubectl exec -it vault-0 -n vault -- sh -c &#39;<br>vault write auth/kubernetes/config \<br>  kubernetes_host=&quot;https://kubernetes.default.svc:443&quot; \<br>  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \<br>  token_reviewer_jwt=&quot;$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot;<br>&#39;</pre><p><strong>How This Works:</strong></p><ol><li>Application pod starts with ServiceAccount token</li><li>Pod sends token to Vault</li><li>Vault validates with Kubernetes API</li><li>Vault returns Vault token to pod</li><li>Pod uses Vault token to access secrets</li></ol><h3>Step 6: Create Transit Encryption Key</h3><pre>kubectl exec -it vault-0 -n vault -- vault write -f transit/keys/tax-calculator</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*Tdeqrq1shNzZildLD7ExZw.png" /></figure><h3>Step 7: Create Access Policy</h3><pre>kubectl exec -it vault-0 -n vault -- vault policy write tax-calculator - &lt;&lt;EOF<br># Read dynamic database credentials<br>path &quot;database/creds/tax-calculator-role&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br># Transit encryption operations<br>path &quot;transit/encrypt/tax-calculator&quot; {<br>  capabilities = [&quot;update&quot;]<br>}<br>path &quot;transit/decrypt/tax-calculator&quot; {<br>  capabilities = [&quot;update&quot;]<br>}<br># Read configuration secrets<br>path &quot;secret/data/config/tax-calculator&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>EOF<br></pre><p><strong>Security Principle: Least Privilege</strong></p><p>This policy grants ONLY:</p><ul><li>Read access to DB creds (not create/delete)</li><li>Encrypt/decrypt operations (not admin functions)</li><li>Read config secrets (not write)</li></ul><h3>Step 8: Create Kubernetes Role</h3><pre>kubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/role/tax-calculator \<br>  bound_service_account_names=tax-calculator \<br>  bound_service_account_namespaces=tax-calculator \<br>  policies=tax-calculator \<br>  ttl=1h</pre><p><strong>What This Means:</strong></p><ul><li>Only pods with ServiceAccount “tax-calculator”</li><li>In namespace “tax-calculator”</li><li>Get the “tax-calculator” policy</li><li>Vault tokens expire after 1 hour</li></ul><h3>Phase 3: Database Setup</h3><h3>Step 1: Deploy PostgreSQL</h3><pre>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: tax-calculator<br>---<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: tax-calculator<br>  namespace: tax-calculator<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: postgres<br>  namespace: tax-calculator<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: tax-calculator<br>      component: database<br>  template:<br>    metadata:<br>      labels:<br>        app: tax-calculator<br>        component: database<br>    spec:<br>      containers:<br>      - name: postgres<br>        image: postgres:15<br>        env:<br>        - name: POSTGRES_DB<br>          value: taxcalc<br>        - name: POSTGRES_USER<br>          value: postgres<br>        - name: POSTGRES_PASSWORD<br>          value: postgres123<br>        ports:<br>        - containerPort: 5432<br>        resources:<br>          requests:<br>            memory: &quot;256Mi&quot;<br>            cpu: &quot;250m&quot;<br>          limits:<br>            memory: &quot;512Mi&quot;<br>            cpu: &quot;500m&quot;<br>        volumeMounts:<br>        - name: postgres-storage<br>          mountPath: /var/lib/postgresql/data<br>          subPath: postgres<br>      volumes:<br>      - name: postgres-storage<br>        emptyDir: {}<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: postgres<br>  namespace: tax-calculator<br>spec:<br>  selector:<br>    app: tax-calculator<br>    component: database<br>  ports:<br>  - port: 5432<br>    targetPort: 5432<br>EOF</pre><p><strong>Wait for PostgreSQL:</strong></p><pre>kubectl wait --for=condition=ready pod -l component=database \<br>  -n tax-calculator --timeout=300s</pre><h3>Step 2: Configure Vault Database Secrets</h3><pre>kubectl exec -it vault-0 -n vault -- vault write database/config/postgres \<br>  plugin_name=postgresql-database-plugin \<br>  allowed_roles=&quot;tax-calculator-role&quot; \<br>  connection_url=&quot;postgresql://{{username}}:{{password}}@postgres.tax-calculator.svc.cluster.local:5432/taxcalc?sslmode=disable&quot; \<br>  username=&quot;postgres&quot; \<br>  password=&quot;postgres123&quot;</pre><h3>Step 3: Create Dynamic Credentials Role</h3><p><strong>PostgreSQL 15 Security Change:</strong></p><p>PostgreSQL 15 removed the default CREATE privilege on public schema. We must explicitly grant it:</p><pre>kubectl exec -it vault-0 -n vault -- vault write database/roles/tax-calculator-role \<br>  db_name=postgres \<br>  creation_statements=&quot;CREATE ROLE \&quot;{{name}}\&quot; WITH LOGIN PASSWORD &#39;{{password}}&#39; VALID UNTIL &#39;{{expiration}}&#39;; \<br>    GRANT ALL PRIVILEGES ON DATABASE taxcalc TO \&quot;{{name}}\&quot;; \<br>    GRANT ALL ON SCHEMA public TO \&quot;{{name}}\&quot;; \<br>    GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \&quot;{{name}}\&quot;; \<br>    GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO \&quot;{{name}}\&quot;; \<br>    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO \&quot;{{name}}\&quot;; \<br>    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO \&quot;{{name}}\&quot;;&quot; \<br>  default_ttl=&quot;1h&quot; \<br>  max_ttl=&quot;24h&quot;</pre><p><strong>Why This Is Powerful:</strong></p><p>Every time the application requests credentials:</p><ol><li>Vault creates a NEW PostgreSQL user</li><li>Grants only necessary permissions</li><li>Sets expiration (1 hour by default)</li><li>Auto-revokes after TTL</li></ol><p><strong>Test it:</strong></p><pre>kubectl exec -it vault-0 -n vault -- vault read database/creds/tax-calculator-role</pre><p><strong>Output:</strong></p><pre>Key                Value<br>---                -----<br>lease_id           database/creds/tax-calculator-role/abc123...<br>lease_duration     1h<br>username           v-root-tax-calc-xyz789...<br>password           A1b2C3d4E5f6G7h8...</pre><p><strong>Try again:</strong></p><pre>kubectl exec -it vault-0 -n vault -- vault read database/creds/tax-calculator-role</pre><p>You’ll get DIFFERENT credentials! That’s dynamic secrets in action.</p><h3>🐳 Phase 4: Container Images</h3><h3>Step 1: Set Environment Variables</h3><pre>export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)<br>export AWS_REGION=&quot;eu-west-2&quot;<br>echo &quot;AWS Account: $AWS_ACCOUNT_ID&quot;<br>echo &quot;AWS Region: $AWS_REGION&quot;</pre><h3>Step 2: Login to ECR</h3><pre>aws ecr get-login-password --region $AWS_REGION | \<br>  docker login --username AWS --password-stdin \<br>  $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com</pre><h3>Step 3: Build &amp; Push Backend</h3><pre>cd tax-calculator-app/backend<br>docker build -t tax-calculator-backend:latest .<br><br>docker tag tax-calculator-backend:latest \<br>  $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/tax-calculator-backend:latest<br><br>docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/tax-calculator-backend:latest</pre><p><strong>Backend Dockerfile Highlights:</strong></p><pre># Multi-stage build for minimal image<br>FROM golang:1.21-alpine AS builder<br>WORKDIR /app<br>COPY . .<br>RUN go build -o main .<br><br>FROM alpine:latest<br>COPY --from=builder /app/main /main<br>CMD [&quot;/main&quot;]</pre><p><strong>Result:</strong> Only ~15MB final image!</p><h3>Step 5: Build &amp; Push Frontend</h3><pre>cd ../frontend<br>docker build -t tax-calculator-frontend:latest .<br><br>docker tag tax-calculator-frontend:latest \<br>  $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/tax-calculator-frontend:latest<br><br>docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/tax-calculator-frontend:latest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*NrWKkmzogPOLLhTtKF22uQ.png" /></figure><h3>☸️ Phase 5: Kubernetes Deployment</h3><h3>Understanding the K8s Architecture</h3><pre>Namespace: tax-calculator<br>├── ServiceAccount (for Vault auth)<br>├── PostgreSQL<br>│   ├── Deployment (1 replica)<br>│   └── Service (ClusterIP)<br>├── Backend<br>│   ├── Deployment (2 replicas)<br>│   └── Service (ClusterIP)<br>└── Frontend<br>    ├── Deployment (2 replicas)<br>    └── Service (LoadBalancer)</pre><h3>Step 1: Backend Deployment</h3><p>Create k8s/backend-deployment.yaml:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: backend<br>  namespace: tax-calculator<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: tax-calculator<br>      component: backend<br>  template:<br>    metadata:<br>      labels:<br>        app: tax-calculator<br>        component: backend<br>    spec:<br>      serviceAccountName: tax-calculator<br>      containers:<br>      - name: backend<br>        image: ${AWS_ACCOUNT_ID}.dkr.ecr.eu-west-2.amazonaws.com/tax-calculator-backend:latest<br>        ports:<br>        - containerPort: 8080<br>        env:<br>        - name: VAULT_ADDR<br>          value: &quot;http://vault.vault.svc.cluster.local:8200&quot;<br>        - name: VAULT_ROLE<br>          value: &quot;tax-calculator&quot;<br>        - name: DATABASE_HOST<br>          value: &quot;postgres.tax-calculator.svc.cluster.local&quot;<br>        - name: DATABASE_PORT<br>          value: &quot;5432&quot;<br>        - name: DATABASE_NAME<br>          value: &quot;taxcalc&quot;<br>        resources:<br>          requests:<br>            memory: &quot;256Mi&quot;<br>            cpu: &quot;250m&quot;<br>          limits:<br>            memory: &quot;512Mi&quot;<br>            cpu: &quot;500m&quot;<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: backend<br>  namespace: tax-calculator<br>spec:<br>  selector:<br>    app: tax-calculator<br>    component: backend<br>  ports:<br>  - port: 8080<br>    targetPort: 8080<br>  type: ClusterIP</pre><h3>Step 2: Frontend with Nginx Reverse Proxy</h3><p><strong>Why Reverse Proxy?</strong></p><p>React apps compile environment variables at BUILD time, not runtime. We can’t use REACT_APP_API_URL dynamically in Kubernetes.</p><p><strong>Solution:</strong> Nginx proxies /api/* to backend:</p><p>Create frontend/nginx.conf:</p><pre>server {<br>    listen 80;<br>    root /usr/share/nginx/html;<br>    index index.html;<br>    <br>    # Serve React app<br>    location / {<br>        try_files $uri $uri/ /index.html;<br>    }<br>    <br>    # Proxy API requests to backend<br>    location /api/ {<br>        proxy_pass http://backend.tax-calculator.svc.cluster.local:8080;<br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>    }<br>    <br>    # Proxy health checks<br>    location /health {<br>        proxy_pass http://backend.tax-calculator.svc.cluster.local:8080/health;<br>    }<br>}</pre><p><strong>Benefits:</strong></p><ul><li>Backend stays internal (ClusterIP only)</li><li>Single LoadBalancer (cost savings)</li><li>No CORS issues</li><li>Production-standard pattern</li></ul><p>Create k8s/frontend-deployment.yaml:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: frontend<br>  namespace: tax-calculator<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: tax-calculator<br>      component: frontend<br>  template:<br>    metadata:<br>      labels:<br>        app: tax-calculator<br>        component: frontend<br>    spec:<br>      containers:<br>      - name: frontend<br>        image: ${AWS_ACCOUNT_ID}.dkr.ecr.eu-west-2.amazonaws.com/tax-calculator-frontend:latest<br>        ports:<br>        - containerPort: 80<br>        resources:<br>          requests:<br>            memory: &quot;128Mi&quot;<br>            cpu: &quot;100m&quot;<br>          limits:<br>            memory: &quot;256Mi&quot;<br>            cpu: &quot;200m&quot;<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: frontend<br>  namespace: tax-calculator<br>  annotations:<br>    service.beta.kubernetes.io/aws-load-balancer-type: &quot;nlb&quot;<br>    service.beta.kubernetes.io/aws-load-balancer-scheme: &quot;internet-facing&quot;<br>spec:<br>  type: LoadBalancer<br>  selector:<br>    app: tax-calculator<br>    component: frontend<br>  ports:<br>  - port: 80<br>    targetPort: 80</pre><p><strong>Why NLB?</strong></p><p>Feature NLB ALB Layer 4 (TCP) 7 (HTTP) Speed Ultra-fast Fast Cost Lower Higher Simplicity Higher Lower</p><p>For simple HTTP, NLB is perfect.</p><h3>Step 3: Deploy Application</h3><pre>cd tax-calculator-app/k8s<br><br># Replace AWS Account ID<br>export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)<br>sed -i &quot;s/\${AWS_ACCOUNT_ID}/$AWS_ACCOUNT_ID/g&quot; *.yaml<br># Deploy<br>kubectl apply -f backend-deployment.yaml<br>kubectl apply -f frontend-deployment.yaml<br># Watch deployment<br>kubectl get pods -n tax-calculator -w</pre><h3>🌐 Phase 6: Access &amp; Testing</h3><h3>Step 1: Get Application URL</h3><pre># Wait for LoadBalancer (2-3 minutes)<br>kubectl get svc frontend -n tax-calculator<br><br># Get URL<br>export APP_URL=$(kubectl get svc frontend -n tax-calculator \<br>  -o jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39;)<br>echo &quot;Application URL: http://$APP_URL&quot;<br></pre><p>You also need to confirm that the status if the Load Balancer is active.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_y9F34GoExTYkuh7d2YNA.png" /></figure><h3>Step 2: Test Endpoints</h3><pre># Health check<br>curl http://$APP_URL/health<br><br># Expected:<br># {&quot;status&quot;:&quot;healthy&quot;,&quot;database&quot;:&quot;healthy&quot;,&quot;vault&quot;:&quot;healthy&quot;}<br># Tax calculation<br>curl -X POST http://$APP_URL/api/v1/calculate \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;income&quot;: 50000,<br>    &quot;national_insurance&quot;: &quot;AB123456C&quot;,<br>    &quot;tax_year&quot;: &quot;2024/2025&quot;<br>  }&#39;<br># Expected:<br># {<br>#   &quot;income&quot;: 50000,<br>#   &quot;income_tax&quot;: 7486,<br>#   &quot;national_insurance_contribution&quot;: 4504.80,<br>#   &quot;take_home&quot;: 38009.20,<br>#   &quot;effective_rate&quot;: 23.98,<br>#   &quot;encrypted_ni&quot;: &quot;vault:v1:8SDd3...&quot;<br># }</pre><h3>Step 3: Open in Browser</h3><p>Use the load balancer dns name or visit http://$APP_URL and you&#39;ll see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*afdNu6AVp0VIsQguyRq6DQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXCZ0HxfD9aZg5Q4xaPG_Q.png" /></figure><h3>🔍 Phase 7: Verify Security</h3><h3>Check Encrypted Data</h3><pre>kubectl exec -it deployment/postgres -n tax-calculator -- \<br>  psql -U postgres -d taxcalc -c \<br>  &quot;SELECT id, income, encrypted_ni FROM calculations LIMIT 5;&quot;</pre><p><strong>Output:</strong></p><pre>id  | income  |           encrypted_ni           <br>-----+---------+----------------------------------<br>  1  |  50000  | vault:v1:8SDd3WHDOjf7Mkb9...<br>  2  |  75000  | vault:v1:9TGe4XIEPkg8Nlc0...</pre><p><strong>No plaintext NI numbers!</strong> ✅</p><h3>Verify Dynamic Credentials</h3><pre># Generate new credentials<br>kubectl exec -it vault-0 -n vault -- \<br>  vault read database/creds/tax-calculator-role<br><br># Check database users<br>kubectl exec -it deployment/postgres -n tax-calculator -- \<br>  psql -U postgres -d taxcalc -c &quot;\du&quot;</pre><p><strong>Output:</strong></p><pre>v-root-tax-calc-abc123... | Valid until 2025-12-23 06:00:00+00<br>v-root-tax-calc-def456... | Valid until 2025-12-23 06:15:00+00</pre><p>Each user has unique credentials with 1-hour expiration! ✅</p><h3>Check Vault UI</h3><pre>kubectl port-forward -n vault svc/vault 8200:8200<br><br><br>### Browser<br>```<br>http://localhost:8200<br>```<br><br>### Login Token<br>```<br>hvs.3UD3jlrVBQZdVOrSpfW2ABCD</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*krzWdjRIW6BmuWc06_Xy7g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFclMKjHS4NY_YkXqfsZHA.png" /></figure><h3>What we have deployed</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*wEjVv7HfhfFtQWuGPjGUBA.png" /></figure><h3>Common Issues &amp; Solutions</h3><h3>Issue 1: Backend CrashLoopBackOff</h3><p><strong>Diagnosis:</strong></p><pre>kubectl logs -n tax-calculator deployment/backend --tail=50</pre><p><strong>Common causes:</strong></p><ul><li>PostgreSQL permission denied (schema issue)</li><li>Can’t connect to Vault</li><li>Missing environment variables</li></ul><p><strong>Solution for PostgreSQL 15:</strong> Already covered in Phase 3, Step 3 (explicit schema grants)</p><h3>Issue 2: LoadBalancer Stuck Pending</h3><p><strong>Diagnosis:</strong></p><pre>kubectl describe svc frontend -n tax-calculator</pre><p><strong>Cause:</strong> Missing subnet tags</p><p><strong>Solution:</strong></p><pre>aws ec2 create-tags \<br>  --resources &lt;subnet-id&gt; \<br>  --tags Key=kubernetes.io/role/elb,Value=1 \<br>  --region eu-west-2</pre><h3>What We’ve Built</h3><h3>Architecture Summary</h3><pre>Internet → NLB → Frontend (Nginx) → Backend (Go) → Vault → PostgreSQL<br>                                      ↓<br>                                    KMS (Auto-unseal)</pre><p><strong>Key Features:</strong></p><p>✅ Dynamic secrets (1-hour TTL)</p><p>✅ Transit encryption for PII</p><p>✅ Zero-trust security model</p><p>✅ High availability (multi-AZ)</p><p>✅ Infrastructure as Code</p><p>✅ Complete audit trail</p><h3>🚀 Next Steps</h3><p><strong>Part 2 of this series covers:</strong></p><ul><li><strong>GitOps</strong>: Automated deployments with ArgoCD</li><li><strong>Observability</strong>: Prometheus, Grafana, and logging</li><li><strong>Enhanced Security</strong>: Network policies, TLS, Pod Security Standards</li></ul><p><strong>Stay tuned!</strong></p><h3>🧹 Cleanup</h3><p>When done:</p><pre># Delete application<br>kubectl delete namespace tax-calculator<br><br># Destroy infrastructure<br>cd terraform/environments/dev<br>terraform destroy -auto-approve</pre><h3>🎉 Conclusion</h3><p>You’ve built a <strong>production-grade application</strong> demonstrating:</p><ul><li>Modern cloud architecture</li><li>Enterprise security practices</li><li>DevOps automation</li><li>Real-world problem solving</li></ul><p><strong>Perfect for:</strong></p><ul><li>Technical interviews</li><li>Portfolio projects</li><li>Learning cloud-native development</li><li>Understanding secrets management</li></ul><p><strong>GitHub:</strong> <a href="https://github.com/tobiakinlade/vault-gitops-platform.git">https://github.com/tobiakinlade/vault-gitops-platform.git</a></p><p><strong>Continue to Part 2 for GitOps, Observability, and Enhanced Security!</strong> 🚀</p><p><strong>Connect with me</strong> on <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">LinkedIn</a> for more AWS, DevOps, and cloud architecture content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9110af7c5fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Production-Grade Observability for AWS EKS:A Complete Step-by-Step Guide (Part 3)]]></title>
            <link>https://medium.com/@tobi.akinlade.co/building-production-grade-observability-for-aws-eks-a-complete-step-by-step-guide-part-3-e3c9668938bc?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3c9668938bc</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Fri, 05 Dec 2025 14:47:47 GMT</pubDate>
            <atom:updated>2025-12-05T14:47:47.096Z</atom:updated>
            <content:encoded><![CDATA[<p>From “Hope It Works” to “I Know It Works” — Monitoring Your Kubernetes Infrastructure Like a Pro</p><h3>Introduction</h3><p>In <a href="https://medium.com/@tobi.akinlade.co/deploying-a-full-stack-application-to-aws-eks-a-complete-kubernetes-guide-10c99335ad52"><strong>Part 1</strong></a>, we deployed the Job Board application to AWS EKS using Terraform.</p><p>In <a href="https://medium.com/@tobi.akinlade.co/building-a-production-grade-kubernetes-ci-cd-pipeline-with-gitops-part-2-d5826056cff3"><strong>Part 2</strong></a>, we built a production-grade CI/CD pipeline powered by GitOps with ArgoCD and GitHub Actions.</p><p>Now, in <strong>Part 3</strong>, we complete the trilogy by adding <strong>production-grade observability</strong> to make our platform fully ready for real-world operations.</p><p><strong>Why Observability Matters:</strong></p><p>Deploying an application is only half the battle. In production, you need to answer:</p><ul><li>Is my application healthy?</li><li>Why is it slow?</li><li>What happened at 3 AM when users reported errors?</li><li>Am I about to run out of resources?</li></ul><p>Without observability, you’re flying blind.</p><p><strong>To follow along with this project, you can clone the repository:</strong> <a href="https://github.com/tobiakinlade/job-board.git">https://github.com/tobiakinlade/job-board.git</a></p><h3>What We’ll Build</h3><p>Component Purpose <strong>Prometheus</strong> Collects and stores metrics from your cluster <strong>Grafana</strong> Visualizes metrics through dashboards <strong>Loki</strong> Aggregates logs from all pods <strong>Promtail</strong> Ships logs from nodes to Loki <strong>Alertmanager</strong> Routes alerts to Slack <strong>ServiceMonitors</strong> Define what Prometheus should scrape <strong>PrometheusRules</strong> Define alerting conditions</p><p><strong>Architecture Overview:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h03D8o08JuH2gQsoPN5SyQ.png" /></figure><p><strong>Prerequisites:</strong> Completed Parts 1 &amp; 2 (EKS cluster with Job Board deployed)</p><p>Also, provide the infrastructure and application, and ensure they are running from where we left off in Part 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JbdYemPDvel5dg5Zi2nnlw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pedsgpBx48EQW25GGCPD-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nngxlk1k6GQGxAoBC6dfVQ.png" /></figure><h3>Step 1: Add Helm Repositories</h3><p>Helm is the package manager for Kubernetes. We’ll use it to install our monitoring stack.</p><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WvB5Z2VqUiqYFw9La-DCWg.png" /></figure><p><strong>Why Helm?</strong> These charts contain battle-tested configurations. Installing Prometheus manually would require dozens of YAML files.</p><h3>Step 2: Create Monitoring Namespace</h3><pre>kubectl create namespace monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FaK6sBmmcMI0K7TIbFYzFQ.png" /></figure><p><strong>Best Practice:</strong> Always isolate monitoring components from application workloads. This prevents resource contention and makes RBAC easier.</p><h3>Step 3: Create Prometheus Configuration</h3><p>Create a directory for monitoring configurations:</p><pre>mkdir -p monitoring</pre><p>Create the Prometheus values file:</p><pre># monitoring/prometheus-values.yaml<br>prometheus:<br>  prometheusSpec:<br>    retention: 7d<br>    resources:<br>      requests:<br>        memory: 256Mi<br>        cpu: 100m<br>      limits:<br>        memory: 1Gi<br>        cpu: 500m<br>    storageSpec: {}<br>    serviceMonitorSelectorNilUsesHelmValues: false<br>    podMonitorSelectorNilUsesHelmValues: false<br>    grafana:<br>  enabled: true<br>  adminPassword: &quot;YourSecurePassword123!&quot;<br>  persistence:<br>    enabled: false<br>  sidecar:<br>    datasources:<br>      enabled: true<br>      defaultDatasourceEnabled: true<br>alertmanager:<br>  enabled: true<br>  alertmanagerSpec:<br>    storage: {}<br>nodeExporter:<br>  enabled: true<br>kubeStateMetrics:<br>  enabled: true<br>prometheusOperator:<br>  resources:<br>    requests:<br>      cpu: 50m<br>      memory: 64Mi<br>    limits:<br>      cpu: 100m<br>      memory: 128Mi</pre><p><strong>Key Settings Explained:</strong></p><ul><li>retention: 7d — Keep metrics for 7 days (adjust based on your needs)</li><li>serviceMonitorSelectorNilUsesHelmValues: false — Allows Prometheus to discover ALL ServiceMonitors, not just those with specific labels</li><li>nodeExporter: enabled — Collects CPU, memory, disk metrics from nodes</li><li>kubeStateMetrics: enabled — Collects Kubernetes object states (pod status, deployment replicas, etc.)</li></ul><h3>Step 4: Install Prometheus Stack</h3><pre>helm install prometheus prometheus-community/kube-prometheus-stack \<br>  --namespace monitoring \<br>  --values monitoring/prometheus-values.yaml \<br>  --wait --timeout 10m</pre><p><strong>What Gets Installed:</strong></p><p>Component Purpose Prometheus Server Scrapes and stores metrics Grafana Web UI for dashboards Alertmanager Handles alert routing Node Exporter Host-level metrics Kube-State-Metrics Kubernetes object metrics Prometheus Operator Manages Prometheus via CRDs</p><p><strong>Verify Installation:</strong></p><pre>kubectl get pods -n monitoring</pre><p>You should see all pods in Running state:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGI79mCSuQ-WV6aHItSfeA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qvpmScFWab2on-EuqNYgA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OJiwhIvrNtEqV_mJRzUAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yOo95F2HXYEUmLfja3qP1w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jo3rN9RXmMcBliYG9Z0rZA.png" /></figure><h3>Step 5: Install Loki for Log Aggregation</h3><p>While Prometheus handles metrics, Loki handles logs. Think of it as “Prometheus for logs.”</p><pre>helm install loki grafana/loki-stack \<br>  --namespace monitoring \<br>  --set loki.persistence.enabled=false \<br>  --set promtail.enabled=true \<br>  --set grafana.enabled=false</pre><p><strong>Why Loki over ELK Stack?</strong></p><ul><li>Lightweight — doesn’t index log content, only labels</li><li>Native Grafana integration</li><li>Same query language as Prometheus (LogQL)</li><li>Lower resource footprint</li></ul><p><strong>Verify:</strong></p><pre>kubectl get pods -n monitoring | grep -E &quot;loki|promtail&quot;</pre><p>Expected output:</p><pre>loki-0                    1/1     Running   0          1m<br>loki-promtail-xxxxx       1/1     Running   0          1m<br>loki-promtail-yyyyy       1/1     Running   0          1m</pre><h3>Step 6: Add Loki as Grafana Datasource</h3><p>Grafana needs to know where to find Loki.</p><pre># monitoring/grafana-datasource-loki.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: grafana-datasource-loki<br>  namespace: monitoring<br>  labels:<br>    grafana_datasource: &quot;1&quot;<br>data:<br>  loki-datasource.yaml: |<br>    apiVersion: 1<br>    datasources:<br>      - name: Loki<br>        type: loki<br>        access: proxy<br>        url: http://loki:3100<br>        isDefault: false</pre><p>Apply and restart Grafana:</p><pre>kubectl apply -f monitoring/grafana-datasource-loki.yaml<br>kubectl rollout restart deployment prometheus-grafana -n monitoring</pre><p><strong>Important:</strong> The label grafana_datasource: &quot;1&quot; triggers Grafana&#39;s sidecar to automatically load this datasource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vua7dAhCPHU9PjsJOT6JKg.png" /></figure><h3>Step 7: Create ServiceMonitors</h3><p>ServiceMonitors are Custom Resources that tell Prometheus what to scrape.</p><pre># monitoring/servicemonitor.yaml<br>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: job-board-backend<br>  namespace: monitoring<br>  labels:<br>    release: prometheus<br>spec:<br>  selector:<br>    matchLabels:<br>      app: backend<br>  namespaceSelector:<br>    matchNames:<br>      - job-board<br>  endpoints:<br>    - port: http<br>      path: /health<br>      interval: 30s<br>---<br>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: job-board-frontend<br>  namespace: monitoring<br>  labels:<br>    release: prometheus<br>spec:<br>  selector:<br>    matchLabels:<br>      app: frontend<br>  namespaceSelector:<br>    matchNames:<br>      - job-board<br>  endpoints:<br>    - port: http<br>      path: /<br>      interval: 30s</pre><p>Apply:</p><pre>kubectl apply -f monitoring/servicemonitor.yaml</pre><p><strong>Critical:</strong> The label release: prometheus must match your Helm release name. Without this, Prometheus won&#39;t discover the ServiceMonitor.</p><p><strong>Verify:</strong></p><pre>kubectl get servicemonitors -n monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yoX-TNw_vAlM5Ul3EUZt5Q.png" /></figure><h3>Step 8: Create Alert Rules</h3><p>PrometheusRules define when to fire alerts.</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: PrometheusRule<br>metadata:<br>  name: job-board-alerts<br>  namespace: monitoring<br>  labels:<br>    release: prometheus<br>spec:<br>  groups:<br>    - name: job-board.rules<br>      rules:<br>        # Pod not ready<br>        - alert: JobBoardPodNotReady<br>          expr: |<br>            kube_pod_status_ready{namespace=&quot;job-board&quot;, condition=&quot;true&quot;} == 0<br>          for: 5m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;Pod {{ $labels.pod }} not ready&quot;<br>            description: &quot;Pod has been not ready for more than 5 minutes&quot;<br><br>        # Pod restarting frequently<br>        - alert: JobBoardPodRestartingFrequently<br>          expr: |<br>            increase(kube_pod_container_status_restarts_total{namespace=&quot;job-board&quot;}[1h]) &gt; 3<br>          for: 5m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;Pod {{ $labels.pod }} restarting frequently&quot;<br>            description: &quot;Pod has restarted {{ $value }} times in the last hour&quot;<br><br>        # High memory usage<br>        - alert: JobBoardHighMemoryUsage<br>          expr: |<br>            (container_memory_usage_bytes{namespace=&quot;job-board&quot;} / container_spec_memory_limit_bytes{namespace=&quot;job-board&quot;}) &gt; 0.85<br>          for: 5m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;High memory usage on {{ $labels.pod }}&quot;<br>            description: &quot;Memory usage is above 85%&quot;<br><br>        # High CPU usage<br>        - alert: JobBoardHighCPUUsage<br>          expr: |<br>            (rate(container_cpu_usage_seconds_total{namespace=&quot;job-board&quot;}[5m]) * 100) &gt; 80<br>          for: 5m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;High CPU usage on {{ $labels.pod }}&quot;<br>            description: &quot;CPU usage is above 80%&quot;<br><br>        # Deployment replicas mismatch<br>        - alert: JobBoardDeploymentReplicasMismatch<br>          expr: |<br>            kube_deployment_spec_replicas{namespace=&quot;job-board&quot;} != kube_deployment_status_replicas_available{namespace=&quot;job-board&quot;}<br>          for: 5m<br>          labels:<br>            severity: critical<br>          annotations:<br>            summary: &quot;Deployment {{ $labels.deployment }} replicas mismatch&quot;<br>            description: &quot;Expected {{ $value }} replicas but some are unavailable&quot;<br><br>        # PVC almost full<br>        - alert: JobBoardPVCAlmostFull<br>          expr: |<br>            (kubelet_volume_stats_used_bytes{namespace=&quot;job-board&quot;} / kubelet_volume_stats_capacity_bytes{namespace=&quot;job-board&quot;}) &gt; 0.85<br>          for: 5m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;PVC {{ $labels.persistentvolumeclaim }} almost full&quot;<br>            description: &quot;PVC is {{ $value | humanizePercentage }} full&quot;<br><br>    - name: job-board.slo<br>      rules:<br>        # SLI: Availability recording rule<br>        - record: job_board:availability:rate5m<br>          expr: |<br>            sum(rate(kube_pod_status_ready{namespace=&quot;job-board&quot;, condition=&quot;true&quot;}[5m]))<br>            / sum(rate(kube_pod_status_ready{namespace=&quot;job-board&quot;}[5m]))<br><br>        # SLO: 99.9% availability target<br>        - alert: JobBoardSLOAvailabilityBreach<br>          expr: |<br>            (1 - job_board:availability:rate5m) &gt; 0.001<br>          for: 5m<br>          labels:<br>            severity: critical<br>            slo: availability<br>          annotations:<br>            summary: &quot;Job Board availability SLO breach&quot;<br>            description: &quot;Availability is below 99.9% target&quot;</pre><p>Apply:</p><pre>kubectl apply -f monitoring/prometheus-rules.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GZMP4t6O1B8k5zc4d7AoXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LOOh2fWm-PSMUVgTy13Dbw.png" /></figure><h3>Step 9: Create Grafana Dashboard</h3><p>Dashboards visualize your metrics in real-time.</p><pre># monitoring/grafana-dashboard.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: job-board-dashboard<br>  namespace: monitoring<br>  labels:<br>    grafana_dashboard: &quot;1&quot;<br>data:<br>  job-board-dashboard.json: |<br>    {<br>      &quot;title&quot;: &quot;Job Board Overview&quot;,<br>      &quot;uid&quot;: &quot;job-board-overview&quot;,<br>      &quot;panels&quot;: [<br>        {<br>          &quot;id&quot;: 1,<br>          &quot;title&quot;: &quot;Pods Running&quot;,<br>          &quot;type&quot;: &quot;stat&quot;,<br>          &quot;gridPos&quot;: {&quot;h&quot;: 4, &quot;w&quot;: 6, &quot;x&quot;: 0, &quot;y&quot;: 0},<br>          &quot;targets&quot;: [{&quot;expr&quot;: &quot;count(kube_pod_status_phase{namespace=\&quot;job-board\&quot;, phase=\&quot;Running\&quot;} == 1)&quot;}],<br>          &quot;fieldConfig&quot;: {<br>            &quot;defaults&quot;: {<br>              &quot;thresholds&quot;: {<br>                &quot;steps&quot;: [<br>                  {&quot;color&quot;: &quot;red&quot;, &quot;value&quot;: null},<br>                  {&quot;color&quot;: &quot;yellow&quot;, &quot;value&quot;: 2},<br>                  {&quot;color&quot;: &quot;green&quot;, &quot;value&quot;: 3}<br>                ]<br>              }<br>            }<br>          }<br>        },<br>        {<br>          &quot;id&quot;: 2,<br>          &quot;title&quot;: &quot;Pod Restarts (1h)&quot;,<br>          &quot;type&quot;: &quot;stat&quot;,<br>          &quot;gridPos&quot;: {&quot;h&quot;: 4, &quot;w&quot;: 6, &quot;x&quot;: 6, &quot;y&quot;: 0},<br>          &quot;targets&quot;: [{&quot;expr&quot;: &quot;sum(increase(kube_pod_container_status_restarts_total{namespace=\&quot;job-board\&quot;}[1h]))&quot;}],<br>          &quot;fieldConfig&quot;: {<br>            &quot;defaults&quot;: {<br>              &quot;thresholds&quot;: {<br>                &quot;steps&quot;: [<br>                  {&quot;color&quot;: &quot;green&quot;, &quot;value&quot;: null},<br>                  {&quot;color&quot;: &quot;yellow&quot;, &quot;value&quot;: 1},<br>                  {&quot;color&quot;: &quot;red&quot;, &quot;value&quot;: 5}<br>                ]<br>              }<br>            }<br>          }<br>        },<br>        {<br>          &quot;id&quot;: 3,<br>          &quot;title&quot;: &quot;CPU Usage Over Time&quot;,<br>          &quot;type&quot;: &quot;timeseries&quot;,<br>          &quot;gridPos&quot;: {&quot;h&quot;: 8, &quot;w&quot;: 12, &quot;x&quot;: 0, &quot;y&quot;: 4},<br>          &quot;targets&quot;: [{&quot;expr&quot;: &quot;rate(container_cpu_usage_seconds_total{namespace=\&quot;job-board\&quot;, container!=\&quot;\&quot;}[5m]) * 100&quot;, &quot;legendFormat&quot;: &quot;{{ pod }}&quot;}],<br>          &quot;fieldConfig&quot;: {&quot;defaults&quot;: {&quot;unit&quot;: &quot;percent&quot;}}<br>        },<br>        {<br>          &quot;id&quot;: 4,<br>          &quot;title&quot;: &quot;Memory Usage Over Time&quot;,<br>          &quot;type&quot;: &quot;timeseries&quot;,<br>          &quot;gridPos&quot;: {&quot;h&quot;: 8, &quot;w&quot;: 12, &quot;x&quot;: 12, &quot;y&quot;: 4},<br>          &quot;targets&quot;: [{&quot;expr&quot;: &quot;container_memory_usage_bytes{namespace=\&quot;job-board\&quot;, container!=\&quot;\&quot;} / 1024 / 1024&quot;, &quot;legendFormat&quot;: &quot;{{ pod }}&quot;}],<br>          &quot;fieldConfig&quot;: {&quot;defaults&quot;: {&quot;unit&quot;: &quot;decmbytes&quot;}}<br>        },<br>        {<br>          &quot;id&quot;: 5,<br>          &quot;title&quot;: &quot;Network Received&quot;,<br>          &quot;type&quot;: &quot;timeseries&quot;,<br>          &quot;gridPos&quot;: {&quot;h&quot;: 8, &quot;w&quot;: 12, &quot;x&quot;: 0, &quot;y&quot;: 12},<br>          &quot;targets&quot;: [{&quot;expr&quot;: &quot;rate(container_network_receive_bytes_total{namespace=\&quot;job-board\&quot;}[5m])&quot;, &quot;legendFormat&quot;: &quot;{{ pod }}&quot;}],<br>          &quot;fieldConfig&quot;: {&quot;defaults&quot;: {&quot;unit&quot;: &quot;Bps&quot;}}<br>        },<br>        {<br>          &quot;id&quot;: 6,<br>          &quot;title&quot;: &quot;Network Transmitted&quot;,<br>          &quot;type&quot;: &quot;timeseries&quot;,<br>          &quot;gridPos&quot;: {&quot;h&quot;: 8, &quot;w&quot;: 12, &quot;x&quot;: 12, &quot;y&quot;: 12},<br>          &quot;targets&quot;: [{&quot;expr&quot;: &quot;rate(container_network_transmit_bytes_total{namespace=\&quot;job-board\&quot;}[5m])&quot;, &quot;legendFormat&quot;: &quot;{{ pod }}&quot;}],<br>          &quot;fieldConfig&quot;: {&quot;defaults&quot;: {&quot;unit&quot;: &quot;Bps&quot;}}<br>        }<br>      ],<br>      &quot;time&quot;: {&quot;from&quot;: &quot;now-1h&quot;, &quot;to&quot;: &quot;now&quot;},<br>      &quot;refresh&quot;: &quot;30s&quot;<br>    }</pre><p>Apply and restart Grafana:</p><pre>kubectl apply -f monitoring/grafana-dashboard.yaml<br>kubectl rollout restart deployment prometheus-grafana -n monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_zy-ELZVRKjWwJKU3PM3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lF6EHlRD0pW2nGNtC0dFAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*33xS3io0smwHBpMWyTJBAQ.png" /></figure><h3>Step 10: Configure Alertmanager for Slack Notifications</h3><p>Now let’s route alerts to Slack so you get notified when things break.</p><p><strong>First, create a Slack Webhook:</strong></p><ol><li>Go to api.slack.com/apps</li><li>Create New App → From Scratch</li><li>Enable Incoming Webhooks</li><li>Add New Webhook to Workspace</li><li>Select channel (e.g., #alerts)</li><li>Copy the Webhook URL</li></ol><p><strong>Create Alertmanager Config:</strong></p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: alertmanager-prometheus-kube-prometheus-alertmanager<br>  namespace: monitoring<br>stringData:<br>  alertmanager.yaml: |<br>    global:<br>      resolve_timeout: 5m<br><br>    route:<br>      group_by: [&#39;alertname&#39;, &#39;namespace&#39;, &#39;severity&#39;]<br>      group_wait: 30s<br>      group_interval: 5m<br>      repeat_interval: 4h<br>      receiver: &#39;slack-receiver&#39;<br>      routes:<br>        - match:<br>            severity: critical<br>          receiver: &#39;slack-receiver&#39;<br>        - match:<br>            severity: warning<br>          receiver: &#39;slack-receiver&#39;<br><br>    receivers:<br>      - name: &#39;slack-receiver&#39;<br>        slack_configs:<br>          - api_url: &#39;https://hooks.slack.com/services/T0C7B7/B0A21V5DXA5/abwCZ8eRtpUka65WN5s2iG3u&#39;<br>            channel: &#39;#general&#39;<br>            send_resolved: true<br>            title: &#39;{{ .GroupLabels.alertname }}&#39;<br>            text: &#39;{{ .CommonAnnotations.summary }}&#39;<br><br>    inhibit_rules:<br>      - source_match:<br>          severity: &#39;critical&#39;<br>        target_match:<br>          severity: &#39;warning&#39;<br>        equal: [&#39;alertname&#39;, &#39;namespace&#39;]</pre><p>Apply:</p><pre>kubectl apply -f monitoring/alertmanager-config.yaml</pre><p><strong>Routing Explained:</strong></p><p>Setting Purpose group_by Group similar alerts together group_wait Wait 30s before sending to batch alerts repeat_interval Don&#39;t repeat same alert for 4 hours inhibit_rules If critical fires, suppress related warnings</p><h3>Step 11: Access Your Monitoring Stack</h3><p><strong>Grafana (Dashboards &amp; Logs):</strong></p><pre>kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80</pre><ul><li>URL: <a href="http://localhost:3000/">http://localhost:3000</a></li><li>Username: admin</li><li>Password: YourSecurePassword123!</li></ul><p><strong>Prometheus (Metrics &amp; Queries):</strong></p><pre>kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090</pre><ul><li>URL: <a href="http://localhost:9090/">http://localhost:9090</a></li></ul><p><strong>Alertmanager (Alert Status):</strong></p><pre>kubectl port-forward svc/prometheus-kube-prometheus-alertmanager -n monitoring 9093:9093</pre><ul><li>URL: <a href="http://localhost:9093/">http://localhost:9093</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ehx7z7ho2jLdIYDuI9bZVQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B2ib5GexkwyW1UX49T4KoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ouQpvXdAcdcyfjfZqGqwhQ.png" /></figure><h3>Verification Checklist</h3><p>Run these commands to verify everything is working:</p><pre># All monitoring pods running?<br>kubectl get pods -n monitoring<br># ServiceMonitors created?<br>kubectl get servicemonitors -n monitoring<br># Alert rules loaded?<br>kubectl get prometheusrules -n monitoring<br># Dashboards and datasources configured?<br>kubectl get configmaps -n monitoring | grep -E &quot;dashboard|datasource&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLA7N5sktPVZsfmBszzfkw.png" /></figure><p><strong>Test Prometheus:</strong></p><p>Open <a href="http://localhost:9090/">http://localhost:9090</a> and run:</p><pre>kube_pod_status_phase{namespace=&quot;job-board&quot;}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LOOh2fWm-PSMUVgTy13Dbw.png" /></figure><p><strong>Test Loki:</strong></p><ol><li>Open Grafana → Explore</li><li>Select “Loki” datasource</li><li>Query: {namespace=&quot;job-board&quot;}</li><li>Click “Run query”</li></ol><p>You should see logs from your Job Board pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PyUabXypUFVicnYl3jFMZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*csCp6gEFWE8WB2JODkb7TQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dJat2tla-ShrxNbilXD13g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DTBe-V3XJJ_MTv713IVifw.png" /></figure><h3>Directory Structure</h3><p>Your monitoring configuration should look like:</p><pre>monitoring/<br>├── prometheus-values.yaml        # Prometheus stack configuration<br>├── grafana-datasource-loki.yaml  # Loki datasource for Grafana<br>├── servicemonitor.yaml           # What to scrape<br>├── prometheus-rules.yaml         # Alert conditions<br>├── grafana-dashboard.yaml        # Dashboard JSON<br>└── alertmanager-config.yaml      # Slack notification config</pre><h3>Troubleshooting Common Issues</h3><p>Issue Solution ServiceMonitor not discovered Ensure release: prometheus label exists Grafana CrashLoopBackOff Check logs: kubectl logs -n monitoring -l app.kubernetes.io/name=grafana Loki not receiving logs Verify Promtail pods: kubectl get pods -n monitoring | grep promtail Alerts not firing Check Prometheus UI → Status → Rules Can&#39;t access dashboards Verify port-forward is running Slack not receiving alerts Test webhook URL with curl</p><h3>Key Takeaways</h3><ol><li><strong>Prometheus</strong> scrapes metrics from your applications at regular intervals</li><li><strong>ServiceMonitors</strong> define what Prometheus should scrape — no config file changes needed</li><li><strong>PrometheusRules</strong> define alerting conditions using PromQL</li><li><strong>Alertmanager</strong> routes alerts based on severity and labels</li><li><strong>Loki</strong> aggregates logs without heavy indexing — perfect for Kubernetes</li><li><strong>Grafana</strong> ties it all together with unified dashboards</li></ol><p><strong>Tip:</strong> Tear down resources when not in use:</p><pre>cd terraform<br>terraform destroy --auto-approve</pre><h3>Conclusion</h3><p>You’ve now built a complete observability stack for your Kubernetes application:</p><p>✅ <strong>Metrics</strong> — Prometheus collecting cluster and application metrics</p><p>✅ <strong>Logs</strong> — Loki aggregating logs from all pods</p><p>✅ <strong>Dashboards</strong> — Grafana visualizing everything in one place</p><p>✅ <strong>Alerts</strong> — Alertmanager routing notifications to Slack</p><p>✅ <strong>GitOps Ready</strong> — All configs stored as YAML in Git</p><p>This completes our 3-part series on deploying a production-grade application to AWS EKS. You now have infrastructure-as-code, automated CI/CD, and full observability — the holy trinity of DevOps.</p><p><strong>Connect with me on </strong><a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/"><strong>LinkedIn</strong></a> <strong>for more AWS, DevOps, and cloud architecture content.</strong></p><p>Happy monitoring! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3c9668938bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Production-Grade Kubernetes CI/CD Pipeline with GitOps (Part 2)]]></title>
            <link>https://medium.com/@tobi.akinlade.co/building-a-production-grade-kubernetes-ci-cd-pipeline-with-gitops-part-2-d5826056cff3?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5826056cff3</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 22:56:06 GMT</pubDate>
            <atom:updated>2025-11-23T22:56:06.483Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Complete Guide to Automated Deployments with GitHub Actions, ArgoCD, and Kustomize</h3><h3>CI/CD High Level Design (HLD)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HjLhGotYaxPMjNgIMwJ8Xw.png" /></figure><h3>EKS Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wG4MBBYYuwAsxZMIk4KLow.png" /></figure><h3>Introduction</h3><p>In <a href="https://medium.com/@tobi.akinlade.co/deploying-a-full-stack-application-to-aws-eks-a-complete-kubernetes-guide-10c99335ad52">Part 1</a>, we deployed the job board application on EKS, and in Part 2, we will implement a CI/CD pipeline using a GitOps approach.</p><p>To follow along with this project, you can clone the repository <a href="https://github.com/tobiakinlade/job-board.git">https://github.com/tobiakinlade/job-board.git</a></p><p>After deploying your EKS cluster with Terraform, you have infrastructure — but how do you deploy applications efficiently? This guide walks you through building a complete CI/CD pipeline using GitOps principles, enabling automated deployments from code commit to production.</p><p><strong>What we’ll build:</strong></p><ul><li>GitOps deployment with ArgoCD</li><li>Kustomize for configuration management</li><li>GitHub Actions for CI/CD automation</li><li>Rollback workflow for incident response</li></ul><h3>1. GitOps Foundation with ArgoCD</h3><h3>Why ArgoCD?</h3><p>Traditional deployments use kubectl apply - error-prone and hard to audit. <strong>GitOps flips this model</strong>: Git becomes the single source of truth, and ArgoCD continuously syncs your cluster to match Git.</p><p><strong>Benefits:</strong></p><ul><li><strong>Declarative:</strong> Git describes desired state</li><li><strong>Auditable:</strong> All changes are Git commits</li><li><strong>Automated:</strong> ArgoCD handles synchronization</li><li><strong>Recoverable:</strong> Easy rollback via Git history</li></ul><h3>Installing ArgoCD</h3><p>After terraform apply creates your EKS cluster, install ArgoCD:</p><p>I have created a script that will provision the infrastructure using terraform, set the kubeconfig context, install ArgoCD, create ArgoCD application</p><pre>### Installing ArgoCD<br><br>I&#39;ve created an automated setup script that handles:<br>1. Infrastructure provisioning with Terraform<br>2. Kubeconfig setup<br>3. ArgoCD installation<br>4. Application creation<br>5. Secrets configuration<br><br>**Quick Start:**<br>```bash<br># Clone the repository<br>git clone https://github.com/tobiakinlade/job-board.git<br>cd job-board<br><br># Run automated setup<br>chmod +x clean-setup.sh<br>./clean-setup.sh<br>```<br><br>The script performs the following steps:<br>- [Show a simplified version with comments]<br>- Full script available at: `scripts/clean-setup.sh`<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stDAFf40jXSCA19qxst4og.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A_D1yZPZAGBA4BW58HCA0g.png" /></figure><p><strong>Wait for ArgoCD pods to be ready</strong> (~2 minutes), then access the UI:</p><pre># Get admin password<br>kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=&quot;{.data.password}&quot; | base64 -d<br><br># Port forward<br>kubectl port-forward svc/argocd-server -n argocd 8080:443<br>```<br>Open `https://localhost:8080` and login with `admin` and the password.<br>**Key ArgoCD Concepts:**<br>- **Application:** Represents your deployed app<br>- **Sync Status:** Git ↔ Cluster comparison<br>- **Health Status:** Pod/service health<br>- **Auto-sync:** Automatically deploys Git changes<br>---<br>## Configuration Management with Kustomize<br>### Why Kustomize Over Helm?<br>While Helm uses templating, **Kustomize uses pure YAML overlays** - easier to read, simpler to maintain, and native to kubectl.<br>**Key advantage:** You keep base manifests clean and layer environment-specific changes on top.<br>### Directory Structure<br>```<br>kubernetes/<br>├── base/                    # Base configurations<br>│   ├── backend-deployment.yaml<br>│   ├── frontend-deployment.yaml<br>│   ├── postgres-statefulset.yaml<br>│   ├── services.yaml<br>│   ├── ingress.yaml<br>│   └── kustomization.yaml<br>└── overlays/<br>    └── dev/                 # Environment-specific<br>        ├── kustomization.yaml<br>        ├── replica-patch.yaml<br>        └── resource-patch.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCFbX7WBhR_sQ_ZG4Y_qDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UmeuzulWLqcR75_IzWVSEA.png" /></figure><h3>Why Overlays?</h3><p><strong>Base</strong> contains your core application definition — deployments, services, ingress. These rarely change.</p><p><strong>Overlays</strong> contain environment-specific customizations:</p><ul><li><strong>Dev:</strong> Single replica, smaller resources, test domain</li><li><strong>Staging:</strong> Multiple replicas, production-like</li><li><strong>Production:</strong> Full scale, monitoring, specific domain</li></ul><p><strong>Example scenario:</strong> Your backend deployment needs 3 replicas in production but 1 in dev. Instead of duplicating the entire deployment manifest, you patch it:</p><p><strong>Base:</strong> backend-deployment.yaml (generic configuration) <strong>Overlay:</strong> replica-patch.yaml sets replicas: 1 for dev</p><p>This keeps configurations DRY (Don’t Repeat Yourself) and makes environment differences explicit.</p><h3>Creating the Base Configuration</h3><p>In kubernetes/base/, create your deployment manifests. Each service gets its own file:</p><ul><li>backend-deployment.yaml - Node.js API</li><li>frontend-deployment.yaml - React SPA</li><li>postgres-statefulset.yaml - Database with persistent storage</li><li>*-service.yaml - Kubernetes services</li><li>ingress.yaml - ALB ingress</li></ul><p>The kustomization.yaml ties them together:</p><pre>apiVersion: kustomize.config.k8s.io/v1beta1<br>kind: Kustomization<br><br>resources:<br>  - namespace.yaml<br>  - configmap.yaml<br>  - postgres-init-configmap.yaml<br>  - postgres-statefulset.yaml<br>  - postgres-service.yaml<br>  - backend-deployment.yaml<br>  - backend-service.yaml<br>  - frontend-deployment.yaml<br>  - frontend-service.yaml<br>  - ingress.yaml<br>  - network-policies.yaml</pre><h3>Creating the Dev Overlay</h3><p>In kubernetes/overlays/dev/kustomization.yaml:</p><pre>apiVersion: kustomize.config.k8s.io/v1beta1<br>kind: Kustomization<br><br>namespace: job-board<br><br>resources:<br>- ../../base<br><br>images:<br>- name: backend-placeholder<br>  newName: 180048382895.dkr.ecr.eu-west-2.amazonaws.com/job-board-backend<br>  newTag: d015f94<br>- name: frontend-placeholder<br>  newName: 180048382895.dkr.ecr.eu-west-2.amazonaws.com/job-board-frontend<br>  newTag: d015f94<br><br>patches:<br>- path: replica-patch.yaml<br>- path: resource-patch.yamlThe magic: images section allows updating container tags without modifying base manifests. Your CI/CD will update newTag automatically.</pre><h3>Connecting ArgoCD to Your Repository</h3><h3>Creating the ArgoCD Application</h3><p>Create argocd/applications.yaml:</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: job-board<br>  namespace: argocd<br>  finalizers:<br>    - resources-finalizer.argocd.argoproj.io<br>spec:<br>  project: default<br>  <br>  source:<br>    repoURL: https://github.com/tobiakinlade/job-board.git<br>    targetRevision: main<br>    path: kubernetes/overlays/dev<br>  <br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: job-board<br>  <br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true<br>      allowEmpty: false<br>    syncOptions:<br>      - CreateNamespace=true<br>    retry:<br>      limit: 5<br>      backoff:<br>        duration: 5s<br>        factor: 2<br>        maxDuration: 3m</pre><p><strong>Key settings explained:</strong></p><ul><li><strong>path:</strong> Points to your overlay (dev/staging/prod)</li><li><strong>automated:</strong> Enables auto-sync (optional)</li><li><strong>prune:</strong> Removes resources deleted from Git</li><li><strong>selfHeal:</strong> Reverts manual cluster changes</li></ul><p>The setup script would apply this script</p><pre>kubectl apply -f argocd/applications.yaml</pre><p>ArgoCD now watches your Git repository. Any change to kubernetes/overlays/dev/ triggers a sync.</p><p><strong>Why this matters:</strong> You’ve just enabled GitOps. No more kubectl apply - just commit to Git and ArgoCD deploys.</p><h3>2. CI/CD Pipeline with GitHub Actions</h3><h3>The Workflow Goal</h3><p>When you push code:</p><ol><li>Build Docker images</li><li>Tag with Git SHA (traceability)</li><li>Push to ECR</li><li>Update Kustomize manifest with new tag</li><li>Commit manifest change</li><li>ArgoCD detects and deploys</li></ol><h3>GitHub Actions Workflow</h3><p>Create .github/workflows/deploy.yml with these jobs:</p><p><strong>Job 1: Build Backend</strong></p><ul><li>Checkout code</li><li>Configure AWS credentials</li><li>Build Docker image from backend/Dockerfile</li><li>Tag: YOUR_ECR/backend:${GITHUB_SHA::7}</li><li>Push to ECR</li></ul><p><strong>Job 2: Build Frontend</strong></p><ul><li>Same as backend but for frontend</li></ul><p><strong>Job 3: Update Manifests</strong></p><ul><li>Update kubernetes/overlays/dev/kustomization.yaml</li><li>Change newTag: to new Git SHA</li><li>Commit and push change</li></ul><p><strong>Why Git SHA tags?</strong></p><ul><li>Every deployment is traceable to exact code</li><li>Easy rollback to any previous SHA</li><li>No “latest” confusion</li></ul><pre>name: Build and Deploy<br><br>on:<br>  push:<br>    branches:<br>      - main<br>    paths:<br>      - &#39;backend/**&#39;<br>      - &#39;frontend/**&#39;<br>      - &#39;kubernetes/**&#39;<br>      - &#39;.github/workflows/deploy.yml&#39;<br>  workflow_dispatch:<br><br>env:<br>  AWS_REGION: eu-west-2<br>  AWS_ACCOUNT_ID: &quot;180048382895&quot;<br><br>jobs:<br>  build-backend:<br>    name: Build Backend<br>    runs-on: ubuntu-latest<br>    outputs:<br>      image-tag: ${{ steps.meta.outputs.image-tag }}<br>    <br>    steps:<br>    - name: Checkout<br>      uses: actions/checkout@v4<br>    <br>    - name: Get short SHA<br>      id: meta<br>      run: |<br>        SHORT_SHA=$(echo ${{ github.sha }} | cut -c1-7)<br>        echo &quot;image-tag=${SHORT_SHA}&quot; &gt;&gt; $GITHUB_OUTPUT<br>    <br>    - name: Setup Node.js<br>      uses: actions/setup-node@v4<br>      with:<br>        node-version: &#39;18&#39;<br>    <br>    - name: Install dependencies<br>      working-directory: backend<br>      run: npm ci<br>    <br>    - name: Run tests<br>      working-directory: backend<br>      run: npm test || echo &quot;Tests not configured&quot;<br>    <br>    - name: Configure AWS credentials<br>      uses: aws-actions/configure-aws-credentials@v4<br>      with:<br>        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>        aws-region: ${{ env.AWS_REGION }}<br>    <br>    - name: Login to ECR<br>      id: login-ecr<br>      uses: aws-actions/amazon-ecr-login@v2<br>    <br>    - name: Build and push<br>      working-directory: backend<br>      env:<br>        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}<br>        IMAGE_TAG: ${{ steps.meta.outputs.image-tag }}<br>      run: |<br>        docker build -t $ECR_REGISTRY/job-board-backend:$IMAGE_TAG .<br>        docker push $ECR_REGISTRY/job-board-backend:$IMAGE_TAG<br>        echo &quot;✅ Pushed backend:$IMAGE_TAG&quot;<br><br>  build-frontend:<br>    name: Build Frontend<br>    runs-on: ubuntu-latest<br>    outputs:<br>      image-tag: ${{ steps.meta.outputs.image-tag }}<br>    <br>    steps:<br>    - name: Checkout<br>      uses: actions/checkout@v4<br>    <br>    - name: Get short SHA<br>      id: meta<br>      run: |<br>        SHORT_SHA=$(echo ${{ github.sha }} | cut -c1-7)<br>        echo &quot;image-tag=${SHORT_SHA}&quot; &gt;&gt; $GITHUB_OUTPUT<br>    <br>    - name: Setup Node.js<br>      uses: actions/setup-node@v4<br>      with:<br>        node-version: &#39;18&#39;<br>    <br>    - name: Install dependencies<br>      working-directory: frontend<br>      run: npm ci<br>    <br>    - name: Run tests<br>      working-directory: frontend<br>      run: npm test -- --passWithNoTests<br>    <br>    - name: Build<br>      working-directory: frontend<br>      run: npm run build<br>    <br>    - name: Configure AWS credentials<br>      uses: aws-actions/configure-aws-credentials@v4<br>      with:<br>        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>        aws-region: ${{ env.AWS_REGION }}<br>    <br>    - name: Login to ECR<br>      id: login-ecr<br>      uses: aws-actions/amazon-ecr-login@v2<br>    <br>    - name: Build and push<br>      working-directory: frontend<br>      env:<br>        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}<br>        IMAGE_TAG: ${{ steps.meta.outputs.image-tag }}<br>      run: |<br>        docker build --build-arg REACT_APP_API_URL=/api -t $ECR_REGISTRY/job-board-frontend:$IMAGE_TAG .<br>        docker push $ECR_REGISTRY/job-board-frontend:$IMAGE_TAG<br>        echo &quot;✅ Pushed frontend:$IMAGE_TAG&quot;<br><br>  update-manifests:<br>    name: Update Kubernetes Manifests<br>    runs-on: ubuntu-latest<br>    needs: [build-backend, build-frontend]<br>    <br>    steps:<br>    - name: Checkout<br>      uses: actions/checkout@v4<br>      with:<br>        token: ${{ secrets.GITHUB_TOKEN }}<br>        fetch-depth: 0<br>    <br>    - name: Setup Kustomize<br>      run: |<br>        curl -s &quot;https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh&quot; | bash<br>        sudo mv kustomize /usr/local/bin/<br>    <br>    - name: Configure Git<br>      run: |<br>        git config user.name &quot;GitHub Actions&quot;<br>        git config user.email &quot;actions@github.com&quot;<br>    <br>    - name: Update images<br>      env:<br>        IMAGE_TAG: ${{ needs.build-backend.outputs.image-tag }}<br>      run: |<br>        # Pull latest to avoid conflicts<br>        git pull origin main --rebase<br>        <br>        cd kubernetes/overlays/dev<br>        <br>        kustomize edit set image \<br>          backend-placeholder=${{ env.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/job-board-backend:${IMAGE_TAG}<br>        <br>        kustomize edit set image \<br>          frontend-placeholder=${{ env.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/job-board-frontend:${IMAGE_TAG}<br>        <br>        cat kustomization.yaml<br>    <br>    - name: Commit and push<br>      env:<br>        IMAGE_TAG: ${{ needs.build-backend.outputs.image-tag }}<br>      run: |<br>        git add kubernetes/overlays/dev/kustomization.yaml<br>        <br>        # Check if there are changes<br>        if git diff --staged --quiet; then<br>          echo &quot;No changes to commit&quot;<br>          exit 0<br>        fi<br>        <br>        git commit -m &quot;🚀 Deploy: ${IMAGE_TAG}<br><br>        Backend: job-board-backend:${IMAGE_TAG}<br>        Frontend: job-board-frontend:${IMAGE_TAG}<br>        <br>        Commit: ${{ github.sha }}<br>        Workflow: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}&quot;<br>        <br>        git push origin main</pre><h3>Required GitHub Secrets</h3><p>Add in repository settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZC_oww3C6F9Oif5Gl8d8gQ.png" /></figure><p>These let GitHub Actions push to your ECR.</p><h3>Testing the Pipeline</h3><p>Let’s see if our pipeline work</p><pre>git add .<br>git commit -m &quot;CI/CD Pipeline&quot;<br>git push origin main</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BtRCrPPNDOqZsMiX5OmuEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYvSpSYn_S7yPzdWEIOHxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-_lSqms-Tsh37rbCGg6BA.png" /></figure><p><strong>What happens:</strong></p><ol><li>GitHub Actions triggers</li><li>Images build</li><li>Images push to ECR</li><li>Manifest updates</li><li>ArgoCD detects change</li><li>ArgoCD syncs to cluster</li><li>Pods roll out</li><li><strong>New version live!</strong> ✅</li></ol><p>Monitor in 3 places:</p><ul><li><strong>GitHub Actions tab:</strong> Build progress</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BtRCrPPNDOqZsMiX5OmuEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYvSpSYn_S7yPzdWEIOHxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-_lSqms-Tsh37rbCGg6BA.png" /></figure><ul><li><strong>kubectl:</strong> watch kubectl get pods -n job-board</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LE_zDhU75qhm-D_I2JFKDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h2M0KjoW082M-Nu7DaM6TA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7iTMTbEI2KAKyLuPphFGg.png" /></figure><ul><li><strong>ArgoCD UI:</strong> Sync status</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IyHyIVXu7CgJ38FtYqkeXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QaZBeo0n_Wet920GUTIBkw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BOiQwNjn8k8DHFakDGYNmA.png" /></figure><p>Let check the application using the ALB</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*37XGZTP6pIIJlVQuQfA3ow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vCN9AFFiWpFi-QBv7eCMuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YTHELfu0EecQSk7g36iLLw.png" /></figure><h3>3. Testing CI/CD</h3><p>Let’s add a new feature to our application to see how Github Actions will continue to integrate changes and ArgoCD will handle the continuous delivery.</p><p>Let’s make some feature changes to the frontend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iCOFZJY5OwHPFdJavYeufg.png" /></figure><p>Commit the change and check how the change will be in sync to ArgoCD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oi5gbix8h_LOm7eXqiPf0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2CWmDAeygGCR0PCrNvgzRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-MUPc1SKcn3i7dooFeOpQ.png" /></figure><h3>4. Rollback Strategy</h3><h3>Two Approaches</h3><p><strong>1. Kubernetes Native (Emergency — 30 seconds)</strong></p><pre>kubectl rollout undo deployment frontend -n job-board</pre><p>Fast, but creates drift — cluster ≠ Git. Use in emergencies, then manually sync Git.</p><p><strong>2. GitOps Rollback (Proper — 3–5 minutes)</strong></p><p>Create .github/workflows/rollback.yml that:</p><ol><li>Accepts a target commit SHA as input</li><li>Extracts image tags from that commit</li><li>Updates kustomization.yaml with old tags</li><li>Commits and pushes</li><li>ArgoCD syncs the rollback</li></ol><p><strong>Why the GitOps approach is better:</strong></p><ul><li><strong>Audit trail:</strong> GitHub issue created automatically</li><li><strong>No drift:</strong> Git remains source of truth</li><li><strong>Team visibility:</strong> Everyone sees the rollback</li><li><strong>Compliance:</strong> Documented reason required</li></ul><pre>name: Rollback Deployment<br><br>on:<br>  workflow_dispatch:<br>    inputs:<br>      component:<br>        description: &#39;Component to rollback&#39;<br>        required: true<br>        type: choice<br>        options:<br>          - backend<br>          - frontend<br>          - both<br>      commit_sha:<br>        description: &#39;Git commit SHA to rollback to (7 characters)&#39;<br>        required: true<br>        type: string<br>      reason:<br>        description: &#39;Reason for rollback&#39;<br>        required: true<br>        type: string<br><br>jobs:<br>  validate:<br>    name: Validate Rollback Target<br>    runs-on: ubuntu-latest<br>    outputs:<br>      is-valid: ${{ steps.check.outputs.valid }}<br>      backend-tag: ${{ steps.check.outputs.backend_tag }}<br>      frontend-tag: ${{ steps.check.outputs.frontend_tag }}<br>    <br>    steps:<br>      - name: Checkout repository<br>        uses: actions/checkout@v4<br>        with:<br>          fetch-depth: 0<br>      <br>      - name: Validate commit exists<br>        id: check<br>        run: |<br>          COMMIT=&quot;${{ inputs.commit_sha }}&quot;<br>          <br>          if ! git cat-file -e ${COMMIT} 2&gt;/dev/null; then<br>            echo &quot;::error::Commit ${COMMIT} not found&quot;<br>            echo &quot;valid=false&quot; &gt;&gt; $GITHUB_OUTPUT<br>            exit 1<br>          fi<br>          <br>          git show ${COMMIT}:kubernetes/overlays/dev/kustomization.yaml &gt; /tmp/old-kustomization.yaml<br>          <br>          BACKEND_TAG=$(grep -A 2 &quot;name: backend-placeholder&quot; /tmp/old-kustomization.yaml | grep &quot;newTag:&quot; | awk &#39;{print $2}&#39;)<br>          FRONTEND_TAG=$(grep -A 2 &quot;name: frontend-placeholder&quot; /tmp/old-kustomization.yaml | grep &quot;newTag:&quot; | awk &#39;{print $2}&#39;)<br>          <br>          echo &quot;valid=true&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;backend_tag=${BACKEND_TAG}&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;frontend_tag=${FRONTEND_TAG}&quot; &gt;&gt; $GITHUB_OUTPUT<br>          <br>          echo &quot;## Rollback Target Validated&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Commit:** ${COMMIT}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Backend Tag:** ${BACKEND_TAG}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Frontend Tag:** ${FRONTEND_TAG}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br><br>  rollback:<br>    name: Perform Rollback<br>    runs-on: ubuntu-latest<br>    needs: validate<br>    if: needs.validate.outputs.is-valid == &#39;true&#39;<br>    <br>    steps:<br>      - name: Checkout repository<br>        uses: actions/checkout@v4<br>      <br>      - name: Update image tags for rollback<br>        run: |<br>          COMPONENT=&quot;${{ inputs.component }}&quot;<br>          BACKEND_TAG=&quot;${{ needs.validate.outputs.backend-tag }}&quot;<br>          FRONTEND_TAG=&quot;${{ needs.validate.outputs.frontend-tag }}&quot;<br>          KUSTOMIZATION=&quot;kubernetes/overlays/dev/kustomization.yaml&quot;<br>          <br>          echo &quot;Current kustomization.yaml:&quot;<br>          cat ${KUSTOMIZATION}<br>          <br>          if [ &quot;${COMPONENT}&quot; = &quot;backend&quot; ] || [ &quot;${COMPONENT}&quot; = &quot;both&quot; ]; then<br>            echo &quot;Rolling back backend to: ${BACKEND_TAG}&quot;<br>            sed -i.bak &quot;/name: backend-placeholder/,/newTag:/ s/newTag: .*/newTag: ${BACKEND_TAG}/&quot; ${KUSTOMIZATION}<br>          fi<br>          <br>          if [ &quot;${COMPONENT}&quot; = &quot;frontend&quot; ] || [ &quot;${COMPONENT}&quot; = &quot;both&quot; ]; then<br>            echo &quot;Rolling back frontend to: ${FRONTEND_TAG}&quot;<br>            sed -i.bak &quot;/name: frontend-placeholder/,/newTag:/ s/newTag: .*/newTag: ${FRONTEND_TAG}/&quot; ${KUSTOMIZATION}<br>          fi<br>          <br>          echo &quot;&quot;<br>          echo &quot;Updated kustomization.yaml:&quot;<br>          cat ${KUSTOMIZATION}<br>      <br>      - name: Commit and push rollback<br>        run: |<br>          git config user.name &quot;GitHub Actions Bot&quot;<br>          git config user.email &quot;actions@github.com&quot;<br>          <br>          git add kubernetes/overlays/dev/kustomization.yaml<br>          <br>          git commit -m &quot;⏮️ Rollback: ${{ inputs.component }} to commit ${{ inputs.commit_sha }}<br>          <br>          Reason: ${{ inputs.reason }}<br>          Actor: ${{ github.actor }}<br>          Component: ${{ inputs.component }}<br>          Target Commit: ${{ inputs.commit_sha }}<br>          <br>          Backend Tag: ${{ needs.validate.outputs.backend-tag }}<br>          Frontend Tag: ${{ needs.validate.outputs.frontend-tag }}&quot;<br>          <br>          git push origin main<br>      <br>      - name: Wait for ArgoCD sync<br>        run: |<br>          echo &quot;⏳ Waiting for ArgoCD to detect changes...&quot;<br>          sleep 30<br>          echo &quot;✅ ArgoCD should now be syncing the rollback&quot;<br>      <br>      - name: Create rollback summary<br>        run: |<br>          echo &quot;## ✅ Rollback Completed&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Component:** ${{ inputs.component }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Target Commit:** ${{ inputs.commit_sha }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Reason:** ${{ inputs.reason }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Actor:** ${{ github.actor }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Backend Tag:** ${{ needs.validate.outputs.backend-tag }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Frontend Tag:** ${{ needs.validate.outputs.frontend-tag }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;### Verify Rollback&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &#39;```bash&#39; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;kubectl get pods -n job-board&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;kubectl get deployment frontend -n job-board -o jsonpath=&#39;{.spec.template.spec.containers[0].image}&#39;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &#39;```&#39; &gt;&gt; $GITHUB_STEP_SUMMARY<br>      <br>      - name: Create rollback issue<br>        uses: actions/github-script@v7<br>        with:<br>          github-token: ${{ secrets.GITHUB_TOKEN }}<br>          script: |<br>            const body = `## 🔄 Rollback Performed\n\n` +<br>                        `**Component:** ${{ inputs.component }}\n` +<br>                        `**Target Commit:** ${{ inputs.commit_sha }}\n` +<br>                        `**Backend Tag:** ${{ needs.validate.outputs.backend-tag }}\n` +<br>                        `**Frontend Tag:** ${{ needs.validate.outputs.frontend-tag }}\n` +<br>                        `**Reason:** ${{ inputs.reason }}\n` +<br>                        `**Actor:** ${{ github.actor }}\n\n` +<br>                        `### Post-Rollback Checklist\n` +<br>                        `- [ ] Verify application health\n` +<br>                        `- [ ] Check logs for errors\n` +<br>                        `- [ ] Investigate root cause\n` +<br>                        `- [ ] Create fix and test\n` +<br>                        `- [ ] Document incident\n\n` +<br>                        `**Workflow:** ${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;<br>            <br>            await github.rest.issues.create({<br>              owner: context.repo.owner,<br>              repo: context.repo.repo,<br>              title: `🔄 Rollback: ${{ inputs.component }} to ${{ inputs.commit_sha }}`,<br>              body: body,<br>              labels: [&#39;rollback&#39;, &#39;incident&#39;]<br>            });</pre><h3>Testing Rollback</h3><p>After deploying your color change, trigger a rollback:</p><ol><li>Find the commit before your change: git log --oneline -10</li><li>Go to GitHub Actions → “Rollback Deployment”</li><li>Enter:</li></ol><ul><li>Component: frontend</li><li>Commit SHA: abc1234 (the old one)</li><li>Reason: “Reverting UI changes — display issues”</li></ul><p>Click “Run workflow”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LiB4zynDeXgOqtC_Lx526g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1SuO-_KMtov-EaaVRbIyg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aupxq57xeuz-v2Bh4aTdOQ.png" /></figure><p><strong>What happens:</strong></p><ul><li>Workflow validates commit exists</li><li>Extracts old image tags</li><li>Updates kustomization.yaml</li><li>ArgoCD syncs</li><li>Old version restored</li><li>GitHub issue created for tracking</li></ul><p><strong>Timeline:</strong> 3–5 minutes to complete rollback and restore service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dNpDFhYq8qq576wwICxK5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Xl93oGcgpzLVzLcVOJUHw.png" /></figure><h3>5. Troubleshooting Common Issues</h3><h3>ArgoCD Won’t Sync</h3><p><strong>Symptom:</strong> Application stuck “OutOfSync”</p><p><strong>Fix:</strong></p><pre># Force sync<br>kubectl patch application job-board -n argocd \<br>  --type merge -p &#39;{&quot;operation&quot;:{&quot;sync&quot;:{}}}&#39;</pre><h3>Pods Not Updating</h3><p><strong>Symptom:</strong> New image pushed but pods still running old version</p><p><strong>Check:</strong></p><ol><li>Did kustomization.yaml update? git log</li><li>Did ArgoCD sync? Check ArgoCD UI</li><li>Does pod spec have new image? kubectl get deployment -o yaml</li></ol><h3>504 Gateway Timeout</h3><p><strong>Symptom:</strong> ALB returns 504 errors</p><p><strong>Cause:</strong> Security groups not allowing ALB → pods</p><p><strong>Fix:</strong> Verify target group health:</p><p>bash</p><pre>aws elbv2 describe-target-groups --region eu-west-2<br>aws elbv2 describe-target-health --target-group-arn YOUR_TG_ARN</pre><p>Targets should be “healthy”. If “unhealthy”, check security group rules allow traffic from ALB to nodes on ports 3000/3001.</p><h3>Conclusion</h3><p>You’ve built a complete CI/CD pipeline using GitOps principles:</p><p>✅ <strong>Infrastructure as Code</strong> — Terraform manages AWS resources</p><p>✅ <strong>GitOps Deployment</strong> — ArgoCD syncs Git → Cluster</p><p>✅ <strong>Configuration Management</strong> — Kustomize handles environments</p><p>✅ <strong>Automated CI/CD</strong> — GitHub Actions builds and deploys</p><p>✅ <strong>Rollback Capability</strong> — Quick recovery from bad deployments</p><p><strong>Key Takeaways:</strong></p><ol><li><strong>Git is the source of truth</strong> — All changes go through Git</li><li><strong>Immutable infrastructure</strong> — Images are immutable, tags change</li><li><strong>Automated sync</strong> — ArgoCD handles deployments</li><li><strong>Audit trail</strong> — Every deployment is traceable</li><li><strong>Easy rollback</strong> — Revert Git commit or use rollback workflow</li></ol><p><strong>What’s Next?</strong></p><p>In <strong>Part 3</strong>, we will implement production-grade monitoring and observability.</p><p>Coming in Part 3:</p><p>📊 <strong>Prometheus &amp; Grafana</strong> — Complete observability stack</p><p>📈 <strong>Custom metrics </strong>— Application-level monitoring</p><p>🔔 <strong>Alerting</strong> — Slack/PagerDuty integrations</p><p>📝 <strong>Log aggregation</strong> — Loki for centralized logging</p><p>🎯 <strong>SLOs &amp; SLIs</strong> — Production-grade reliability</p><p><strong>Connect with me</strong> on <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">LinkedIn</a> for more AWS, DevOps, and cloud architecture content.</p><p>Happy deploying! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5826056cff3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Full-Stack Application to AWS EKS: A Complete Kubernetes Guide]]></title>
            <link>https://medium.com/@tobi.akinlade.co/deploying-a-full-stack-application-to-aws-eks-a-complete-kubernetes-guide-10c99335ad52?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/10c99335ad52</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Wed, 05 Nov 2025 15:57:15 GMT</pubDate>
            <atom:updated>2025-11-05T15:57:15.966Z</atom:updated>
            <content:encoded><![CDATA[<h3>Series Overview</h3><p>Welcome to this comprehensive guide on deploying a production-ready full-stack application to AWS EKS (Elastic Kubernetes Service). This series will take you from zero to a live, scalable application running on Kubernetes in AWS.</p><p><strong>What You’ll Build:</strong></p><ul><li>A React frontend serving a job board interface</li><li>A Node.js/Express backend API</li><li>A PostgreSQL database with persistent storage</li><li>All running on AWS EKS with professional-grade infrastructure</li></ul><p><strong>What You’ll Learn:</strong></p><ul><li>How Kubernetes resources connect and depend on each other</li><li>The critical order of deployment for Kubernetes manifests</li><li>AWS EKS integration with Load Balancers and storage</li><li>Network security with Calico policies</li><li>Infrastructure as Code with Terraform</li><li>Production deployment best practices</li></ul><h3>High-Level Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*f-E4Z5hY_3F2efBZ69-fzg.png" /></figure><h3>AWS Resources Created</h3><ul><li><strong>VPC</strong>: Custom networking with public and private subnets across multiple AZs</li><li><strong>EKS Cluster</strong>: Managed Kubernetes control plane</li><li><strong>EC2 Node Group</strong>: Auto-scaling worker nodes (2–5 instances)</li><li><strong>ECR Repositories</strong>: Private Docker image storage</li><li><strong>Application Load Balancer</strong>: Internet-facing entry point</li><li><strong>EBS Volumes</strong>: Persistent storage for the database</li><li><strong>IAM Roles</strong>: Secure access permissions</li></ul><h3>1. Prerequisites and Setup</h3><h3>Required Tools</h3><p>Before starting, ensure you have these tools installed:</p><p><strong>AWS CLI</strong> — For AWS operations</p><p><strong>kubectl</strong> — Kubernetes command-line tool</p><p><strong>Terraform</strong> — Infrastructure as Code tool</p><h3>2. Clone the Repository</h3><pre>git clone https://github.com/tobiakinlade/job-board.git<br>cd job-board</pre><p>The repository structure you’ll work with:</p><pre>job-board/<br>├── frontend/          # React application<br>├── backend/           # Node.js API<br>├── terraform/         # Infrastructure code<br>└── kubernetes/        # Kubernetes manifests<br>    ├── namespace.yaml<br>    ├── configmap.yaml<br>    ├── secrets.yaml<br>    ├── postgres-init-configmap.yaml<br>    ├── pvc.yaml<br>    ├── postgres-statefulset.yaml<br>    ├── postgres-service.yaml<br>    ├── backend-deployment.yaml<br>    ├── backend-service.yaml<br>    ├── frontend-deployment.yaml<br>    ├── ingress.yaml<br>    └── network-policies.yaml</pre><p>Run the application application locally</p><pre>cd to the job-board directory<br><br>docker compose up</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*idd46ZR6VCxYNpdXyzgcxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eKLv-AYt4KfunN8P3TQAVA.png" /></figure><h3>3. Understanding the Kubernetes Resource Dependencies</h3><p>This is the <strong>most critical concept</strong> in Kubernetes deployment. Resources have dependencies — some must exist before others can be created.</p><h3>The Dependency Chain</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*ntaa3Vn2Kbf0CA6LkWMJew.png" /></figure><h3>Why This Order Matters</h3><p><strong>Namespace First:</strong> All resources live in a namespace. Without it, nothing else can be created in that namespace.</p><p><strong>ConfigMaps &amp; Secrets Before Deployments:</strong> Deployments reference these for configuration. If they don’t exist, pods will fail to start with “ConfigMapNotFound” or “SecretNotFound” errors.</p><p><strong>PVC Before StatefulSet:</strong> The database needs persistent storage. The PVC must exist so the StatefulSet can claim it.</p><p><strong>Database Before Backend:</strong> The backend connects to the database on startup. If the database service doesn’t exist, backend pods will crash with connection errors.</p><p><strong>Services After Deployments:</strong> While services can technically be created first, creating them after deployments is clearer. Services select pods by labels — no pods, no endpoints.</p><p><strong>Ingress Last:</strong> The Ingress routes traffic to services. Those services must exist with ready endpoints before the ingress can route traffic.</p><p><strong>Network Policies Last:</strong> These restrict traffic between pods. Apply them after everything works to avoid blocking legitimate traffic during setup.</p><h3>Key Kubernetes Concepts</h3><h4>1. Namespace Isolation</h4><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: job-board</pre><ul><li>Creates a logical boundary for resources</li><li>Enables resource quotas and access control</li><li>All subsequent resources will use namespace: job-board</li></ul><h4>2. ConfigMaps (Non-Sensitive Configuration)</h4><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: job-board-config<br>  namespace: job-board<br>data:<br>  DB_HOST: &quot;postgres-service&quot;      # ← Points to the database service name<br>  DB_PORT: &quot;5432&quot;<br>  DB_NAME: &quot;jobboard&quot;</pre><p><strong>Connection Point:</strong> Backend deployment references this:</p><pre>env:<br>- name: DB_HOST<br>  valueFrom:<br>    configMapKeyRef:<br>      name: job-board-config        # ← Must match ConfigMap name<br>      key: DB_HOST                  # ← Must match a key in the ConfigMap</pre><h4>3. Secrets (Sensitive Data)</h4><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: job-board-secrets<br>  namespace: job-board<br>type: Opaque<br>data:<br>  DB_USER: am9iYm9hcmQ=             # ← Base64 encoded &quot;jobboard&quot;<br>  DB_PASSWORD: cGFzc3dvcmQ=         # ← Base64 encoded &quot;password&quot;</pre><p><strong>Connection Point:</strong> Deployments reference secrets similarly to ConfigMaps:</p><pre>env:<br>- name: DB_PASSWORD<br>  valueFrom:<br>    secretKeyRef:<br>      name: job-board-secrets       # ← Must match Secret name<br>      key: DB_PASSWORD              # ← Must match a key in the Secret</pre><h4>4. PersistentVolumeClaim (Storage Request)</h4><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: postgres-pvc<br>  namespace: job-board<br>spec:<br>  accessModes:<br>    - ReadWriteOnce                 # Single node can mount<br>  resources:<br>    requests:<br>      storage: 10Gi                 # Request 10Gi<br>  storageClassName: gp2             # AWS EBS GP2 type</pre><p><strong>What Happens:</strong></p><ul><li>On AWS EKS, this automatically provisions an EBS volume</li><li>The volume persists even if the pod is deleted</li><li>The StatefulSet will claim this storage</li></ul><h4>5. StatefulSet (Database)</h4><pre>apiVersion: apps/v1<br>kind: StatefulSet<br>metadata:<br>  name: postgres<br>  namespace: job-board<br>spec:<br>  serviceName: postgres-service     # ← Must match Service name (defines network identity)<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: postgres                 # ← This label connects to the Service<br>  template:<br>    metadata:<br>      labels:<br>        app: postgres               # ← Must match selector above<br>    spec:<br>      containers:<br>      - name: postgres<br>        image: postgres:15-alpine<br>        volumeMounts:<br>        - name: postgres-storage<br>          mountPath: /var/lib/postgresql/data<br>        env:<br>        - name: POSTGRES_DB<br>          valueFrom:<br>            configMapKeyRef:<br>              name: job-board-config  # ← References ConfigMap<br>              key: DB_NAME<br>        - name: POSTGRES_USER<br>          valueFrom:<br>            secretKeyRef:<br>              name: job-board-secrets # ← References Secret<br>              key: DB_USER<br>      volumes:<br>      - name: postgres-storage<br>        persistentVolumeClaim:<br>          claimName: postgres-pvc   # ← Must match PVC name</pre><p><strong>Why StatefulSet and not Deployment:</strong></p><ul><li>Stable network identity (postgres-0, postgres-1, etc.)</li><li>Ordered deployment and scaling</li><li>Persistent storage per pod</li></ul><h4>6. Service (Database Network Access)</h4><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: postgres-service            # ← This is the DNS name other pods use<br>  namespace: job-board<br>spec:<br>  type: ClusterIP                   # Internal only (not accessible from internet)<br>  clusterIP: None                   # Headless service for StatefulSet<br>  selector:<br>    app: postgres                   # ← Must match StatefulSet pod labels<br>  ports:<br>  - port: 5432<br>    targetPort: 5432<br>    protocol: TCP</pre><p><strong>How It Connects:</strong></p><ul><li>Service creates a DNS entry: postgres-service.job-board.svc.cluster.local</li><li>Backend pods can connect using just postgres-service (same namespace)</li><li>Kubernetes load balances across pods matching the selector</li></ul><h4>7. Deployment (Backend Application)</h4><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: backend<br>  namespace: job-board<br>spec:<br>  replicas: 2                       # 2 instances for high availability<br>  selector:<br>    matchLabels:<br>      app: backend                  # ← Must match template labels<br>  template:<br>    metadata:<br>      labels:<br>        app: backend                # ← This label connects to Service<br>    spec:<br>      containers:<br>      - name: backend<br>        image: &lt;YOUR_ECR_URL&gt;/backend:latest<br>        ports:<br>        - containerPort: 3001       # Port the app listens on<br>        env:<br>        - name: DB_HOST<br>          valueFrom:<br>            configMapKeyRef:<br>              name: job-board-config  # ← Must exist before deployment<br>              key: DB_HOST<br>        # ... more environment variables</pre><p><strong>Connection Points:</strong></p><ul><li>Reads from ConfigMap and Secret (must exist first)</li><li>Connects to postgres-service via DB_HOST</li><li>Exposes port 3001 for the backend Service</li></ul><h4>8. Service (Backend API Access)</h4><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: backend-service             # ← Frontend will connect to this<br>  namespace: job-board<br>spec:<br>  type: ClusterIP                   # Internal only<br>  selector:<br>    app: backend                    # ← Matches Backend Deployment labels<br>  ports:<br>  - port: 3001<br>    targetPort: 3001<br>    protocol: TCP</pre><p><strong>How It Works:</strong></p><ul><li>Creates DNS: backend-service.job-board.svc.cluster.local</li><li>Load balances traffic across both backend pods</li><li>Ingress will route /api/* traffic here</li></ul><h4>9. Deployment (Frontend Application)</h4><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: frontend<br>  namespace: job-board<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: frontend<br>  template:<br>    metadata:<br>      labels:<br>        app: frontend               # ← Connects to Ingress (if NodePort Service exists)<br>    spec:<br>      containers:<br>      - name: frontend<br>        image: &lt;YOUR_ECR_URL&gt;/frontend:latest<br>        ports:<br>        - containerPort: 3000<br>        env:<br>        - name: REACT_APP_API_URL<br>          value: &quot;/api&quot;             # Relative path - ALB will route to backend</pre><p><strong>Note:</strong> Frontend doesn’t need a Service unless you’re using NodePort. With ALB Ingress, traffic goes directly to pods.</p><h4>10. Ingress (ALB Configuration)</h4><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: job-board-ingress<br>  namespace: job-board<br>  annotations:<br>    kubernetes.io/ingress.class: alb<br>    alb.ingress.kubernetes.io/scheme: internet-facing<br>    alb.ingress.kubernetes.io/target-type: ip<br>spec:<br>  rules:<br>  - http:<br>      paths:<br>      - path: /api/*                # Backend API routes<br>        pathType: Prefix<br>        backend:<br>          service:<br>            name: backend-service   # ← Must match Backend Service name<br>            port:<br>              number: 3001<br>      - path: /*                    # All other routes (frontend)<br>        pathType: Prefix<br>        backend:<br>          service:<br>            name: backend-service   # Routes to backend, which serves frontend static files<br>            port:<br>              number: 3001</pre><p><strong>What This Creates:</strong></p><ul><li>An AWS Application Load Balancer</li><li>Path-based routing (API vs frontend)</li><li>Public URL to access your application</li><li>Takes 2–3 minutes to provision</li></ul><p><strong>Connection Points:</strong></p><ul><li>References backend-service by name</li><li>Service must exist with ready endpoints</li><li>ALB provisions and registers pod IPs as targets</li></ul><h4>11. Network Policies (Security Layer)</h4><pre>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: postgres-network-policy<br>  namespace: job-board<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: postgres                 # ← Apply to postgres pods<br>  policyTypes:<br>  - Ingress<br>  ingress:<br>  - from:<br>    - podSelector:<br>        matchLabels:<br>          app: backend              # ← Only backend pods can connect<br>    ports:<br>    - protocol: TCP<br>      port: 5432</pre><p><strong>How Policies Connect:</strong></p><ul><li>Use label selectors to target specific pods</li><li>Define what traffic is allowed (everything else is blocked)</li><li>Backend policy allows frontend to reach it</li><li>Frontend policy allows internet (ALB) to reach it</li><li>Creates a “zero-trust” network within Kubernetes</li></ul><h3>Visual Connection Map</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*RazOm7y6FpqdZGdmhz5wgQ.png" /></figure><h3>4. Infrastructure Provisioning with Terraform</h3><p>Terraform creates all the AWS infrastructure your Kubernetes cluster needs. Understanding what it provisions helps you understand the environment your application runs in.</p><h3>What Terraform Provisions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*4SeAL6Gq28JRy9M16cKRQg.png" /></figure><h3>Networking Layer (VPC)</h3><p><strong>What it creates:</strong></p><ul><li>Custom VPC with CIDR block 10.0.0.0/16</li><li>Public subnets across multiple Availability Zones (for load balancers and NAT gateways)</li><li>Private subnets across multiple AZs (for EKS worker nodes)</li><li>Internet Gateway for outbound internet access</li><li>NAT Gateway(s) to allow private subnet instances to reach the internet</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Security</strong>: Worker nodes in private subnets aren’t directly accessible from the internet</li><li><strong>High Availability</strong>: Multi-AZ deployment ensures your application survives AZ failures</li><li><strong>Isolation</strong>: Your Kubernetes cluster runs in its own isolated network environment</li><li><strong>Best Practice</strong>: Follows AWS Well-Architected Framework for network design</li></ul><h3>EKS Cluster (Managed Kubernetes Control Plane)</h3><p><strong>What it creates:</strong></p><ul><li>Managed Kubernetes control plane (API server, etcd, scheduler, controller manager)</li><li>Node group with 2–10 t3.medium EC2 instances (configurable)</li><li>Auto-scaling configuration based on workload demands</li><li>Cluster security groups for pod-to-pod communication</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Fully Managed</strong>: AWS handles control plane updates, patching, and high availability</li><li><strong>Scalability</strong>: Auto-scaling node groups adapt to application load</li><li><strong>Security</strong>: Integrated with AWS IAM for authentication and authorization</li><li><strong>Production-Ready</strong>: Enterprise-grade Kubernetes without operational overhead</li><li><strong>IRSA Support</strong>: IAM Roles for Service Accounts enable fine-grained permissions for pods</li></ul><h3>ECR (Elastic Container Registry)</h3><p><strong>What it creates:</strong></p><ul><li>Private Docker repositories for backend and frontend images</li><li>Image scanning on push for vulnerability detection</li><li>Lifecycle policies to retain only the last 10 tagged images</li><li>Encryption at rest (AES256)</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Security</strong>: Private registries with IAM-based access control</li><li><strong>Integration</strong>: Seamless integration with EKS (no Docker Hub rate limits)</li><li><strong>Cost Optimization</strong>: Automatic cleanup of old images</li><li><strong>Compliance</strong>: Built-in vulnerability scanning for security compliance</li></ul><h3>IAM Roles &amp; IRSA (IAM Roles for Service Accounts)</h3><p><strong>What it creates:</strong></p><ul><li><strong>EBS CSI Driver Role</strong>: Allows the CSI driver to create/attach/detach EBS volumes</li><li><strong>Load Balancer Controller Role</strong>: Enables automatic ALB provisioning and management</li><li><strong>Node Role</strong>: Permits worker nodes to pull images from ECR and interact with AWS services</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Least Privilege</strong>: Each component gets only the permissions it needs</li><li><strong>No Static Credentials</strong>: Uses temporary credentials via OIDC federation</li><li><strong>Security Best Practice</strong>: Follows AWS security recommendations for EKS</li><li><strong>Audit Trail</strong>: All AWS API calls are logged via CloudTrail</li></ul><h3>EBS CSI Driver (Persistent Storage)</h3><p><strong>What it creates:</strong></p><ul><li>Helm release deploying the AWS EBS CSI driver</li><li>Storage class configured for gp3 volumes (faster and cheaper than gp2)</li><li>Dynamic volume provisioning capability</li><li>Integration with Kubernetes PVC (PersistentVolumeClaim)</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Data Persistence</strong>: Database data survives pod restarts and rescheduling</li><li><strong>Dynamic Provisioning</strong>: Volumes are created automatically when requested</li><li><strong>Performance</strong>: gp3 volumes offer better baseline performance than gp2</li><li><strong>Cost Effective</strong>: Pay only for what you use; volumes scale independently</li></ul><h3>AWS Load Balancer Controller</h3><p><strong>What it creates:</strong></p><ul><li>Helm release deploying the AWS ALB Ingress Controller</li><li>Automatic ALB provisioning when Ingress resources are created</li><li>Target group registration for pod IPs</li><li>Health check configuration</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Automatic Load Balancing</strong>: Creates and configures ALBs declaratively via Kubernetes Ingress</li><li><strong>Native AWS Integration</strong>: Uses Application Load Balancer (Layer 7) for advanced routing</li><li><strong>Path-Based Routing</strong>: Routes /api/* to backend and / to frontend</li><li><strong>Health Checks</strong>: Ensures traffic only goes to healthy pods</li><li><strong>Cost Optimization</strong>: Uses IP target mode (no NodePort overhead)</li></ul><h3>Security Groups</h3><p><strong>What it creates:</strong></p><ul><li>ALB security group (allows HTTP/HTTPS from internet)</li><li>Rules allowing ALB to communicate with pods on specific ports (3000, 3001)</li><li>Rules for both cluster and node security groups (covers all scenarios)</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Defense in Depth</strong>: Multiple layers of security (AWS + Kubernetes Network Policies)</li><li><strong>Precise Access Control</strong>: Only necessary ports are open</li><li><strong>Pod-Level Security</strong>: Works with CNI to protect pod communication</li><li><strong>Separation of Concerns</strong>: Different rules for public-facing vs. internal traffic</li></ul><h3>Kubernetes Resources (Managed by Terraform)</h3><p><strong>What it creates:</strong></p><ul><li><strong>Namespace</strong>: Logical isolation for application resources</li><li><strong>Ingress</strong>: Declaratively provisions ALB with routing rules</li><li><strong>Storage Class</strong>: Default storage class for dynamic PVC provisioning</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Infrastructure as Code</strong>: Kubernetes resources version-controlled alongside infrastructure</li><li><strong>Consistency</strong>: Same deployment process across environments</li><li><strong>Automation</strong>: Fully automated setup with no manual kubectl commands needed</li></ul><h3>S3 Backend &amp; State Locking</h3><p><strong>What it creates:</strong></p><ul><li>S3 bucket for storing Terraform state</li><li>DynamoDB table for state locking</li></ul><p><strong>Why it’s important:</strong></p><ul><li><strong>Team Collaboration</strong>: Shared state enables multiple team members to run Terraform</li><li><strong>State Locking</strong>: Prevents concurrent modifications that could corrupt infrastructure</li><li><strong>History</strong>: S3 versioning provides audit trail of infrastructure changes</li><li><strong>Disaster Recovery</strong>: State can be restored from S3 backups</li></ul><h3>Deployment Steps</h3><h4>Initialize Terraform</h4><pre>cd terraform<br>terraform init</pre><p>This downloads AWS provider plugins and initializes the backend.</p><h4>Review the Plan</h4><pre>terraform plan</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfJRk1vl7CG1lsn0iUJtaA.png" /></figure><h4>Apply Infrastructure</h4><pre>terraform apply --auto-approve</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*my-o95jItbVdn1zgaXozPA.png" /></figure><h4>Configure kubectl</h4><p>Connect your kubectl to the new EKS cluster:</p><pre>aws eks update-kubeconfig --region &lt;YOUR_REGION&gt; --name $(terraform output -raw cluster_name)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*EDlGb0kfj2sfCWyi24_dkQ.png" /></figure><p>Verify connection:</p><pre>kubectl get nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jOhDDllLzTdy-kYZjNsd0A.png" /></figure><p>You should see 2 nodes in “Ready” status.</p><h3>5. Container Registry Setup (ECR)</h3><p>Before deploying to Kubernetes, you need to build your Docker images and push them to ECR.</p><h3>Why ECR?</h3><ul><li>Private, secure image storage</li><li>Integrated with IAM (no Docker Hub credentials needed)</li><li>Fast pulls from within AWS</li><li>Automatic image scanning for vulnerabilities</li></ul><h3>Build and Push Images</h3><pre>#!/bin/bash<br><br>set -e<br><br>VERSION=${1:-&quot;latest&quot;}<br>AWS_REGION=${AWS_REGION:-&quot;eu-west-2&quot;}<br>AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)<br>CLUSTER_NAME=&quot;job-board-eks&quot;<br><br>BACKEND_ECR=&quot;${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CLUSTER_NAME}/backend&quot;<br>FRONTEND_ECR=&quot;${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${CLUSTER_NAME}/frontend&quot;<br><br>echo &quot;=======================================&quot;<br>echo &quot;  Push Docker Images to ECR&quot;<br>echo &quot;=======================================&quot;<br>echo &quot;Region: ${AWS_REGION}&quot;<br>echo &quot;Version: ${VERSION}&quot;<br>echo &quot;&quot;<br><br># Login to ECR<br>echo &quot;Logging in to ECR...&quot;<br>aws ecr get-login-password --region ${AWS_REGION} | \<br>  docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com<br><br># Build images<br>echo &quot;Building backend image...&quot;<br>docker build -t job-board-backend:${VERSION} ./backend<br><br>echo &quot;Building frontend image...&quot;<br>docker build -t job-board-frontend:${VERSION} ./frontend<br><br># Tag images<br>echo &quot;Tagging images...&quot;<br>docker tag job-board-backend:${VERSION} ${BACKEND_ECR}:${VERSION}<br>docker tag job-board-backend:${VERSION} ${BACKEND_ECR}:latest<br>docker tag job-board-frontend:${VERSION} ${FRONTEND_ECR}:${VERSION}<br>docker tag job-board-frontend:${VERSION} ${FRONTEND_ECR}:latest<br><br># Push images<br>echo &quot;Pushing backend image...&quot;<br>docker push ${BACKEND_ECR}:${VERSION}<br>docker push ${BACKEND_ECR}:latest<br><br>echo &quot;Pushing frontend image...&quot;<br>docker push ${FRONTEND_ECR}:${VERSION}<br>docker push ${FRONTEND_ECR}:latest<br><br>echo &quot;&quot;<br>echo &quot;✅ Images pushed successfully!&quot;<br>echo &quot;Backend:  ${BACKEND_ECR}:${VERSION}&quot;<br>echo &quot;Frontend: ${FRONTEND_ECR}:${VERSION}&quot;<br><br># Save to env file<br>cat &gt; .env.eks &lt;&lt; EOF<br>BACKEND_IMAGE=${BACKEND_ECR}:${VERSION}<br>FRONTEND_IMAGE=${FRONTEND_ECR}:${VERSION}<br>AWS_REGION=${AWS_REGION}<br>AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID}<br>EOF<br><br>echo &quot;Environment variables saved to .env.eks&quot;</pre><pre># use the push-to-ecr.sh and make it executable<br><br>chmod +x push-to-ecr.sh<br><br># run the script<br><br>./push-to-ecr.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9tlLhhxKoTdZTpGeHe6fHw.png" /></figure><h3>6. Kubernetes Deployment</h3><p>Now for the main event — deploying your application.</p><p>Instead of applying the Kubernetes file one by one, we can automate the deployment script.</p><pre>cd to the application folder<br><br>execute the deploy-to-eks.sh script</pre><pre>#!/bin/bash<br><br>set -e<br><br>echo &quot;=======================================&quot;<br>echo &quot;  Deploy to AWS EKS&quot;<br>echo &quot;=======================================&quot;<br>echo &quot;&quot;<br><br># Step 1: Push images to ECR<br>echo &quot;Step 1: Building and pushing images to ECR...&quot;<br>./push-to-ecr.sh<br><br># Step 2: Load environment variables<br>source .env.eks<br><br># Step 3: Update manifests<br>echo &quot;&quot;<br>echo &quot;Step 2: Updating Kubernetes manifests...&quot;<br>sed -i &#39;&#39; &quot;s|REPLACE_WITH_ECR_BACKEND_URL:latest|${BACKEND_IMAGE}|g&quot; kubernetes/backend-deployment.yaml<br>sed -i &#39;&#39; &quot;s|REPLACE_WITH_ECR_FRONTEND_URL:latest|${FRONTEND_IMAGE}|g&quot; kubernetes/frontend-deployment.yaml<br><br>echo &quot;✓ Manifests updated&quot;<br><br># Step 4: Configure kubectl<br>echo &quot;&quot;<br>echo &quot;Step 3: Configuring kubectl...&quot;<br>aws eks update-kubeconfig --region eu-west-2 --name job-board-eks<br>echo &quot;✓ kubectl configured&quot;<br><br># Step 5: Deploy to Kubernetes<br>echo &quot;&quot;<br>echo &quot;Step 4: Deploying to EKS...&quot;<br>kubectl apply -f kubernetes/namespace.yaml<br>kubectl apply -f kubernetes/configmap.yaml<br>kubectl apply -f kubernetes/secrets.yaml<br>kubectl apply -f kubernetes/pvc.yaml<br>kubectl apply -f kubernetes/postgres-init-configmap.yaml<br>kubectl apply -f kubernetes/postgres-statefulset.yaml<br>kubectl apply -f kubernetes/postgres-service.yaml<br><br>echo &quot;Waiting for PostgreSQL...&quot;<br>kubectl wait --for=condition=ready pod -l app=postgres -n job-board --timeout=300s<br><br>kubectl apply -f kubernetes/backend-deployment.yaml<br>kubectl apply -f kubernetes/backend-service.yaml<br><br>echo &quot;Waiting for backend...&quot;<br>kubectl wait --for=condition=available deployment/backend -n job-board --timeout=300s<br><br>kubectl apply -f kubernetes/frontend-deployment.yaml<br>kubectl apply -f kubernetes/frontend-service.yaml<br><br>echo &quot;Waiting for frontend...&quot;<br>kubectl wait --for=condition=available deployment/frontend -n job-board --timeout=300s<br><br>kubectl apply -f kubernetes/ingress.yaml<br><br>echo &quot;&quot;<br>echo &quot;✅ Deployment complete!&quot;<br>echo &quot;&quot;<br>echo &quot;Getting Application Load Balancer URL...&quot;<br>sleep 30<br><br>ALB_URL=$(kubectl get ingress job-board-ingress -n job-board -o jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39; 2&gt;/dev/null || echo &quot;&quot;)<br><br>if [ ! -z &quot;$ALB_URL&quot; ]; then<br>    echo &quot;🎉 Application URL: http://${ALB_URL}&quot;<br>else<br>    echo &quot;⏳ ALB is still provisioning. Check status with:&quot;<br>    echo &quot;   kubectl get ingress job-board-ingress -n job-board&quot;<br>fi<br><br>echo &quot;&quot;<br>echo &quot;Useful commands:&quot;<br>echo &quot;  kubectl get all -n job-board&quot;<br>echo &quot;  kubectl logs -f -l app=backend -n job-board&quot;<br>echo &quot;  kubectl logs -f -l app=frontend -n job-board&quot;</pre><p><strong>Verify in AWS Console:</strong></p><ul><li>EC2 → Load Balancers → Find your ALB</li><li>Target Groups → See registered pod IPs</li><li>Health checks should show “healthy”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2aS9dAhjQ3ghMCn-laydQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qX9gf3KnNGIir3u6bjuXlw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*D-Scbv4tuCVG1EFahqc1zQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxWJe8caDjy0027QTKdv-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFUw2x1bdN2MyYY1UnuqjQ.png" /></figure><h3>7. Cleanup</h3><p>When you’re done testing or want to stop incurring charges, proper cleanup is essential.</p><h3>Cleanup Order (Important!)</h3><h3>🧾 Step-by-Step Cleanup Order</h3><h4>Connect to AWS and verify cluster exists</h4><pre>aws eks list-clusters --region eu-west-2</pre><h4>Delete Kubernetes resources</h4><p>If the cluster still exists:</p><pre>aws eks update-kubeconfig --region eu-west-2 --name job-board-eks<br>kubectl delete namespace job-board --ignore-not-found</pre><p>This deletes your workloads and automatically removes the ALB created by the Ingress Controller.<br> If the ALB still exists afterward, delete it manually in the AWS Console → EC2 → Load Balancers.</p><h4>(Optional) Remove orphaned Kubernetes resources from Terraform state</h4><p>If the cluster was deleted manually:</p><pre>terraform state rm helm_release.aws_load_balancer_controller<br>terraform state rm helm_release.ebs_csi_driver<br>terraform state rm kubernetes_namespace.job_board<br>terraform state rm kubernetes_ingress_v1.job_board<br>terraform state rm kubernetes_storage_class_v1.ebs_gp3</pre><p>This stops Terraform from trying to talk to a non-existent cluster.</p><h4>Destroy the Terraform infrastructure</h4><p>Now safely tear down AWS resources (EKS, VPC, IAM, etc.):</p><pre>cd terraform<br>terraform destroy -auto-approve</pre><p>Terraform will destroy:</p><ul><li>EKS cluster (if still present)</li><li>Node groups</li><li>IAM roles / policies</li><li>Security groups</li><li>VPC / subnets</li><li>ECR repositories (force delete the container image)</li><li>DynamoDB table and S3 backend bucket (if included)</li></ul><h4>Verify cleanup</h4><pre>aws eks list-clusters --region eu-west-2<br>aws ecr describe-repositories --region eu-west-2<br>aws ec2 describe-load-balancers --region eu-west-2<br>aws cloudformation list-stacks --region eu-west-2</pre><h3>Troubleshooting Common Issues</h3><h3>Issue 1: Pods in CrashLoopBackOff</h3><p><strong>Symptoms:</strong></p><pre>kubectl get pods -n job-board<br># Shows: backend-xxxxx   0/1   CrashLoopBackOff</pre><p><strong>Diagnosis:</strong></p><pre>kubectl describe pod &lt;pod-name&gt; -n job-board<br>kubectl logs &lt;pod-name&gt; -n job-board<br>kubectl logs &lt;pod-name&gt; -n job-board --previous  # Last crash logs</pre><p><strong>Common Causes:</strong></p><p><strong>1. Can’t Connect to Database</strong></p><ul><li><strong>Error:</strong> ECONNREFUSED or getaddrinfo ENOTFOUND</li><li><strong>Fix:</strong> Check DB_HOST in ConfigMap matches service name exactly</li></ul><pre>kubectl get svc -n job-board  # Service should be &quot;postgres-service&quot;<br>kubectl get configmap job-board-config -n job-board -o yaml  # Check DB_HOST value</pre><p><strong>2. Wrong Secret Values</strong></p><ul><li><strong>Error:</strong> authentication failed</li><li><strong>Fix:</strong> Verify base64 encoded values</li></ul><pre>kubectl get secret job-board-secrets -n job-board -o jsonpath=&#39;{.data.DB_PASSWORD}&#39; | base64 -d<br># Should print your actual password</pre><p><strong>3. Image Pull Error</strong></p><ul><li><strong>Error:</strong> ErrImagePull or ImagePullBackOff</li><li><strong>Fix:</strong> Check ECR authentication and image URL</li></ul><pre># Re-authenticate Docker to ECR<br>aws ecr get-login-password --region &lt;YOUR_REGION&gt; | docker login --username AWS --password-stdin &lt;ECR_URL&gt;<br><br># Verify image exists in ECR<br>aws ecr describe-images --repository-name job-board-eks/backend --region &lt;YOUR_REGION&gt;</pre><p><strong>Common Causes:</strong></p><p><strong>1. ALB Not Healthy</strong></p><ul><li>Go to AWS Console → EC2 → Target Groups</li><li>Check registered targets health status</li><li>If unhealthy, check:</li><li>Health check path exists (/api/health)</li><li>Security groups allow ALB → Pods</li><li>Pods are actually listening on the port</li></ul><p><strong>2. Service Selector Mismatch</strong></p><ul><li>Service selector must match pod labels exactly</li></ul><pre># Check service selector<br>kubectl get svc backend-service -n job-board -o yaml | grep selector -A 2<br><br># Check pod labels<br>kubectl get pods -n job-board --show-labels</pre><h3>Conclusion</h3><p>Congratulations! You’ve successfully:</p><p>✅ Built a full-stack application<br>✅ Containerised it with Docker<br>✅ Provisioned AWS infrastructure with Terraform<br>✅ Deployed to a production Kubernetes cluster<br>✅ Configured networking and load balancing<br>✅ Implemented security with Network Policies<br>✅ Tested high availability and scaling<br>✅ Understood resource dependencies and ordering</p><h3><strong>Want more?</strong> Follow the series for upcoming articles on:</h3><ul><li>Part 2: CI/CD with GitHub Actions</li><li>Part 3: Monitoring with Prometheus/Grafana</li></ul><p><strong>Connect with me</strong> on <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">LinkedIn</a> for more AWS, DevOps, and cloud architecture content.</p><p>Happy deploying! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10c99335ad52" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Production-Grade Observability for AWS ECS: A Complete Step-by-Step Guide]]></title>
            <link>https://medium.com/@tobi.akinlade.co/building-production-grade-observability-for-aws-ecs-a-complete-step-by-step-guide-a7e32aa3ff01?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7e32aa3ff01</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Sat, 25 Oct 2025 00:56:16 GMT</pubDate>
            <atom:updated>2025-10-25T00:56:16.435Z</atom:updated>
            <content:encoded><![CDATA[<h3>From “Hope It Works” to “I Know It Works” — Monitoring Your Infrastructure Like a Pro</h3><p><em>Part 3 of the AWS ECS Production Deployment Series</em></p><h3><strong>The Wake-Up Call</strong></h3><p>It was 2 AM. My phone buzzed. Then rang.</p><p>“The app’s down. Customers can’t log in.”</p><p>Half-asleep, I scrambled for my laptop. How long had it been down? What broke? When did it start?<br> No answers. No metrics. No alerts. Just silence — until an angry tweet broke it.</p><p>That night I learned: you can’t manage what you can’t measure. And you can’t fix what you don’t know is broken.</p><p>In <a href="https://medium.com/@tobi.akinlade.co/automating-aws-ecs-deployment-part-1-infrastructure-as-code-with-terraform-9ce9e69d6436">Part 1</a>, we deployed a containerised app on AWS ECS. In <a href="https://medium.com/@tobi.akinlade.co/automating-aws-ecs-deployment-part-2-ci-cd-pipeline-with-github-actions-66082cc70f17">Part 2</a>, we built CI/CD automation.<br> But we were still flying blind.</p><p><strong>This article changes that.</strong></p><p>By the end, you’ll have a production observability stack that tells you:</p><ul><li>✅ When something breaks (before customers notice)</li><li>✅ What broke (specific metrics and logs)</li><li>✅ Why it broke (historical context)</li><li>✅ Who changed what (drift detection)</li><li>✅ How to fix it (actionable alerts)</li></ul><h3>Prerequisites</h3><p>From Parts 1 &amp; 2, you should have:</p><p>✅ AWS ECS cluster with running application<br>✅ Application Load Balancer<br>✅ RDS database<br>✅ Terraform managing infrastructure<br>✅ GitHub Actions for CI/CD</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7KDgCm2JZl2w5IlDdCakhg.png" /></figure><h3>Picking up from Part 2</h3><p>To continue from where we stop, manually run the terraform apply workflow on Github action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3_s-0qGJH18rjwU2H4oIQQ.png" /></figure><p>I have updated the Deploy to ECS workflow to send a email notification of the deployment using AWS SES.</p><pre>name: Deploy to Dev<br><br>on:<br>  push:<br>    branches:<br>      - main<br>    paths:<br>      - &#39;app/**&#39;<br>      - &#39;Dockerfile&#39;<br>      - &#39;requirements.txt&#39;<br>  workflow_dispatch:<br><br>env:<br>  AWS_REGION: eu-west-2<br>  ECR_REPOSITORY: attendance-app-dev<br>  ECS_CLUSTER: attendance-app-dev-cluster<br>  ECS_SERVICE: attendance-app-dev-service<br>  CONTAINER_NAME: attendance-app-container<br><br>jobs:<br>  test:<br>    name: Run Tests<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Checkout<br>        uses: actions/checkout@v4<br><br>      - name: Set up Python<br>        uses: actions/setup-python@v5<br>        with:<br>          python-version: &#39;3.11&#39;<br>          cache: &#39;pip&#39;<br><br>      - name: Install and Test<br>        run: |<br>          pip install -r requirements.txt<br>          pip install flake8 pytest<br>          flake8 app --count --select=E9,F63,F7,F82 --show-source<br>          echo &quot;✅ Tests passed&quot;<br><br>  deploy:<br>    name: Deploy to Dev<br>    runs-on: ubuntu-latest<br>    needs: test<br>    permissions:<br>      id-token: write<br>      contents: read<br>    outputs:<br>      image: ${{ steps.build.outputs.image }}<br>      status: ${{ job.status }}<br>      deployment_time: ${{ steps.timestamp.outputs.time }}<br>    <br>    steps:<br>      - uses: actions/checkout@v4<br>      <br>      - name: Record Deployment Start Time<br>        id: timestamp<br>        run: |<br>          TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)<br>          echo &quot;time=$TIMESTAMP&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;📅 Deployment started at: $TIMESTAMP&quot;<br>      <br>      - name: Configure AWS<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br>          aws-region: ${{ env.AWS_REGION }}<br><br>      - name: Login to ECR<br>        id: login-ecr<br>        uses: aws-actions/amazon-ecr-login@v2<br><br>      - name: Build and Push<br>        id: build<br>        env:<br>          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}<br>        run: |<br>          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:${{ github.sha }} .<br>          docker tag $ECR_REGISTRY/$ECR_REPOSITORY:${{ github.sha }} $ECR_REGISTRY/$ECR_REPOSITORY:latest<br>          docker push $ECR_REGISTRY/$ECR_REPOSITORY:${{ github.sha }}<br>          docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest<br>          echo &quot;image=$ECR_REGISTRY/$ECR_REPOSITORY:${{ github.sha }}&quot; &gt;&gt; $GITHUB_OUTPUT<br><br>      - name: Update ECS<br>        run: |<br>          aws ecs describe-task-definition --task-definition attendance-app-dev --query taskDefinition &gt; task-def.json<br>          <br>      - name: Render Task Definition<br>        id: task-def<br>        uses: aws-actions/amazon-ecs-render-task-definition@v1<br>        with:<br>          task-definition: task-def.json<br>          container-name: ${{ env.CONTAINER_NAME }}<br>          image: ${{ steps.build.outputs.image }}<br><br>      - name: Deploy to ECS<br>        uses: aws-actions/amazon-ecs-deploy-task-definition@v1<br>        with:<br>          task-definition: ${{ steps.task-def.outputs.task-definition }}<br>          service: ${{ env.ECS_SERVICE }}<br>          cluster: ${{ env.ECS_CLUSTER }}<br>          wait-for-service-stability: true<br><br>      - name: Record Deployment Metrics<br>        if: success()<br>        run: |<br>          TIMESTAMP=&quot;${{ steps.timestamp.outputs.time }}&quot;<br>          VERSION=&quot;${{ github.sha }}&quot;<br>          ACTOR=&quot;${{ github.actor }}&quot;<br>          <br>          echo &quot;📊 Recording deployment metrics in CloudWatch...&quot;<br>          <br>          # 1. Record deployment count<br>          aws cloudwatch put-metric-data \<br>            --namespace &quot;AttendanceApp/Deployments&quot; \<br>            --metric-name &quot;DeploymentCount&quot; \<br>            --value 1 \<br>            --dimensions Environment=dev,Status=success \<br>            --timestamp $TIMESTAMP \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          # 2. Record deployment event<br>          aws cloudwatch put-metric-data \<br>            --namespace &quot;AttendanceApp/Events&quot; \<br>            --metric-name &quot;DeploymentEvent&quot; \<br>            --value 1 \<br>            --dimensions Environment=dev,DeployedBy=$ACTOR,Version=$VERSION \<br>            --timestamp $TIMESTAMP \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          # 3. Record deployment duration<br>          START_TIME=$(date -d &quot;$TIMESTAMP&quot; +%s)<br>          END_TIME=$(date +%s)<br>          DURATION=$((END_TIME - START_TIME))<br>          <br>          aws cloudwatch put-metric-data \<br>            --namespace &quot;AttendanceApp/Deployments&quot; \<br>            --metric-name &quot;DeploymentDuration&quot; \<br>            --value $DURATION \<br>            --unit Seconds \<br>            --dimensions Environment=dev \<br>            --timestamp $(date -u +%Y-%m-%dT%H:%M:%SZ) \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          echo &quot;✅ Deployment metrics recorded:&quot;<br>          echo &quot;   • Deployment count: 1&quot;<br>          echo &quot;   • Duration: ${DURATION}s&quot;<br>          echo &quot;   • Version: $VERSION&quot;<br><br>      - name: Record Failed Deployment<br>        if: failure()<br>        run: |<br>          TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)<br>          <br>          aws cloudwatch put-metric-data \<br>            --namespace &quot;AttendanceApp/Deployments&quot; \<br>            --metric-name &quot;DeploymentCount&quot; \<br>            --value 1 \<br>            --dimensions Environment=dev,Status=failure \<br>            --timestamp $TIMESTAMP \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          echo &quot;❌ Failed deployment recorded in CloudWatch&quot;<br><br>      - name: Summary<br>        run: |<br>          echo &quot;### ✅ Deployed Successfully!&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Image:** ${{ steps.build.outputs.image }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**By:** ${{ github.actor }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Time:** ${{ steps.timestamp.outputs.time }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;📊 **Metrics:** Recorded in CloudWatch&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br><br>  notify:<br>    name: Send Email Notification<br>    needs: deploy<br>    if: always()<br>    runs-on: ubuntu-latest<br>    permissions:<br>      id-token: write<br>      contents: read<br>    <br>    steps:<br>      - name: Configure AWS<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br>          aws-region: ${{ env.AWS_REGION }}<br><br>      - name: Send Deployment Notification<br>        run: |<br>          STATUS=&quot;${{ needs.deploy.result }}&quot;<br>          <br>          # Set email content based on deployment status<br>          if [ &quot;$STATUS&quot; == &quot;success&quot; ]; then<br>            EMOJI=&quot;✅&quot;<br>            SUBJECT=&quot;Deployment Successful - Dev Environment&quot;<br>            TITLE=&quot;Deployment Successful&quot;<br>            COLOR=&quot;#4CAF50&quot;<br>            MESSAGE=&quot;Your code has been successfully deployed to the development environment.&quot;<br>          else<br>            EMOJI=&quot;❌&quot;<br>            SUBJECT=&quot;Deployment Failed - Dev Environment&quot;<br>            TITLE=&quot;Deployment Failed&quot;<br>            COLOR=&quot;#f44336&quot;<br>            MESSAGE=&quot;The deployment failed. Please check the logs and try again.&quot;<br>          fi<br>          <br>          # Create HTML email<br>          cat &gt; email.html &lt;&lt; &#39;EOF&#39;<br>          &lt;!DOCTYPE html&gt;<br>          &lt;html&gt;<br>          &lt;body style=&quot;font-family: Arial, sans-serif; margin: 0; padding: 0; background-color: #f4f4f4;&quot;&gt;<br>            &lt;div style=&quot;max-width: 600px; margin: 40px auto; background: white; border-radius: 8px; box-shadow: 0 2px 4px rgba(0,0,0,0.1); overflow: hidden;&quot;&gt;<br>              <br>              &lt;!-- Header --&gt;<br>              &lt;div style=&quot;background: COLOR_PLACEHOLDER; color: white; padding: 40px 20px; text-align: center;&quot;&gt;<br>                &lt;h1 style=&quot;margin: 0; font-size: 28px;&quot;&gt;EMOJI_PLACEHOLDER TITLE_PLACEHOLDER&lt;/h1&gt;<br>                &lt;p style=&quot;margin: 15px 0 0 0; font-size: 16px; opacity: 0.95;&quot;&gt;Dev Environment&lt;/p&gt;<br>              &lt;/div&gt;<br>              <br>              &lt;!-- Content --&gt;<br>              &lt;div style=&quot;padding: 40px 30px;&quot;&gt;<br>                &lt;p style=&quot;font-size: 16px; color: #333; line-height: 1.6; margin-top: 0;&quot;&gt;<br>                  MESSAGE_PLACEHOLDER<br>                &lt;/p&gt;<br>                <br>                &lt;!-- Deployment Details --&gt;<br>                &lt;div style=&quot;background: #f8f9fa; padding: 20px; border-radius: 6px; margin: 25px 0; border-left: 4px solid COLOR_PLACEHOLDER;&quot;&gt;<br>                  &lt;h3 style=&quot;margin: 0 0 15px 0; color: #333; font-size: 16px;&quot;&gt;📋 Deployment Details&lt;/h3&gt;<br>                  &lt;table style=&quot;width: 100%; border-collapse: collapse;&quot;&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Deployed by:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px;&quot;&gt;ACTOR_PLACEHOLDER&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Time:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px;&quot;&gt;TIME_PLACEHOLDER&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Commit:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px; font-family: monospace;&quot;&gt;COMMIT_PLACEHOLDER&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Branch:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px;&quot;&gt;BRANCH_PLACEHOLDER&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Image:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px; word-break: break-all;&quot;&gt;IMAGE_PLACEHOLDER&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                  &lt;/table&gt;<br>                &lt;/div&gt;<br>                <br>                &lt;!-- Action Buttons --&gt;<br>                &lt;div style=&quot;text-align: center; margin: 35px 0 20px 0;&quot;&gt;<br>                  &lt;a href=&quot;https://app.tech-with-tobi.com&quot; <br>                     style=&quot;display: inline-block; background: #2196F3; color: white; padding: 14px 28px; text-decoration: none; border-radius: 6px; margin: 0 5px; font-weight: 500; font-size: 15px;&quot;&gt;<br>                    🌐 View App<br>                  &lt;/a&gt;<br>                  &lt;a href=&quot;https://github.com/REPO_PLACEHOLDER/actions/runs/RUN_ID_PLACEHOLDER&quot; <br>                     style=&quot;display: inline-block; background: #607D8B; color: white; padding: 14px 28px; text-decoration: none; border-radius: 6px; margin: 0 5px; font-weight: 500; font-size: 15px;&quot;&gt;<br>                    📊 View Logs<br>                  &lt;/a&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>              <br>              &lt;!-- Footer --&gt;<br>              &lt;div style=&quot;background: #f8f9fa; padding: 20px 30px; text-align: center; border-top: 1px solid #dee2e6;&quot;&gt;<br>                &lt;p style=&quot;margin: 0; font-size: 13px; color: #6c757d;&quot;&gt;<br>                  🤖 Automated deployment notification from GitHub Actions<br>                &lt;/p&gt;<br>                &lt;p style=&quot;margin: 8px 0 0 0; font-size: 12px; color: #adb5bd;&quot;&gt;<br>                  Attendance App CI/CD Pipeline<br>                &lt;/p&gt;<br>              &lt;/div&gt;<br>              <br>            &lt;/div&gt;<br>          &lt;/body&gt;<br>          &lt;/html&gt;<br>          EOF<br>          <br>          # Replace placeholders with actual values<br>          sed -i &quot;s|COLOR_PLACEHOLDER|${COLOR}|g&quot; email.html<br>          sed -i &quot;s|EMOJI_PLACEHOLDER|${EMOJI}|g&quot; email.html<br>          sed -i &quot;s|TITLE_PLACEHOLDER|${TITLE}|g&quot; email.html<br>          sed -i &quot;s|MESSAGE_PLACEHOLDER|${MESSAGE}|g&quot; email.html<br>          sed -i &quot;s|ACTOR_PLACEHOLDER|${{ github.actor }}|g&quot; email.html<br>          sed -i &quot;s|TIME_PLACEHOLDER|$(date -u &#39;+%Y-%m-%d %H:%M:%S UTC&#39;)|g&quot; email.html<br>          sed -i &quot;s|COMMIT_PLACEHOLDER|${{ github.sha }}|g&quot; email.html<br>          sed -i &quot;s|BRANCH_PLACEHOLDER|${{ github.ref_name }}|g&quot; email.html<br>          sed -i &quot;s|IMAGE_PLACEHOLDER|${{ needs.deploy.outputs.image }}|g&quot; email.html<br>          sed -i &quot;s|REPO_PLACEHOLDER|${{ github.repository }}|g&quot; email.html<br>          sed -i &quot;s|RUN_ID_PLACEHOLDER|${{ github.run_id }}|g&quot; email.html<br>          <br>          # Convert HTML to JSON-safe string<br>          BODY_JSON=$(jq -Rs . &lt; email.html)<br>          <br>          # Send email via AWS SES<br>          aws ses send-email \<br>            --from &quot;tobi.akinlade.co@gmail.com&quot; \<br>            --destination &quot;ToAddresses=${{ secrets.EMAIL_RECIPIENTS }}&quot; \<br>            --message &quot;{<br>              \&quot;Subject\&quot;: {<br>                \&quot;Data\&quot;: \&quot;${SUBJECT}\&quot;,<br>                \&quot;Charset\&quot;: \&quot;UTF-8\&quot;<br>              },<br>              \&quot;Body\&quot;: {<br>                \&quot;Html\&quot;: {<br>                  \&quot;Data\&quot;: ${BODY_JSON},<br>                  \&quot;Charset\&quot;: \&quot;UTF-8\&quot;<br>                }<br>              }<br>            }&quot; \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          # Clean up<br>          rm -f email.html<br>          <br>          if [ &quot;$STATUS&quot; == &quot;success&quot; ]; then<br>            echo &quot;✅ Success notification sent to team&quot;<br>          else<br>            echo &quot;❌ Failure notification sent to team&quot;<br>          fi<br><br>      - name: Notification Summary<br>        run: |<br>          STATUS=&quot;${{ needs.deploy.result }}&quot;<br>          <br>          if [ &quot;$STATUS&quot; == &quot;success&quot; ]; then<br>            echo &quot;### ✅ Deployment &amp; Notification Complete!&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;📧 Success email sent to: \`${{ secrets.EMAIL_RECIPIENTS }}\`&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;📊 Metrics recorded in CloudWatch&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          else<br>            echo &quot;### ⚠️ Deployment Failed - Team Notified&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;📧 Failure email sent to: \`${{ secrets.EMAIL_RECIPIENTS }}\`&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;📊 Failure metrics recorded in CloudWatch&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          fi</pre><p><strong>You’ll need to add new repository secret</strong></p><pre>secret: EMAIL_FROM, value: noreply@tech-with-tobi.com<br>secret: EMAIL_RECIPIENTS, value: sam@gmail.com,emmanuel@gmail.com<br>secret: EMAIL_RECIPIENTS_URGENT, value: sam@gmail.com,emmanuel@gmail.com</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P3gFc8yCG-sFj6GyXxfGvg.png" /></figure><p>Save the new deploy.yml and commit to git, it will run the pipeline and you should have a running application and also receive a notification about the deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJ-_3XKTOL8I0sYpRRn95w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v4k6_vIiEFtI5HrSDIrmMQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnQdKJZSSvYCQfarQ7ag7w.png" /></figure><h3>The Metrics That Matter</h3><p><strong>1. ECS CPU &amp; Memory</strong></p><ul><li><strong>Shows:</strong> Average CPU and memory usage.</li><li><strong>Why:</strong> High CPU (&gt;80%) or memory (&gt;80%) = overload or OOM risk.</li><li><strong>Watch for:</strong></li><li>Gradual memory rise → leak</li><li>Sudden spikes → traffic/code issue</li><li>Constantly high → under-provisioned</li></ul><p><strong>2. Running Tasks Count</strong></p><ul><li><strong>Shows:</strong> Running vs desired ECS tasks.</li><li><strong>Why:</strong> Gaps = task failures or restarts; 0 = outage.</li><li><strong>Watch for:</strong> Frequent restarts or fewer running tasks.</li></ul><p><strong>3. ALB Request Metrics</strong></p><ul><li><strong>Shows:</strong> Request count &amp; response time.</li><li><strong>Why:</strong> Reflects traffic and performance.</li><li><strong>Watch for:</strong></li><li>Response time &gt;2s → performance issue</li><li>Requests drop → outage/DNS issue</li><li>Traffic spikes → surge or attack</li></ul><p><strong>4. Target Health</strong></p><ul><li><strong>Shows:</strong> Healthy vs unhealthy ALB targets.</li><li><strong>Why:</strong> Unhealthy = lost capacity or outage.</li><li><strong>Watch for:</strong> Any unhealthy targets or flapping status.</li></ul><p><strong>5. RDS Metrics</strong></p><ul><li><strong>Shows:</strong> DB CPU and connections.</li><li><strong>Why:</strong> DB often limits performance.</li><li><strong>Watch for:</strong></li><li>CPU &gt;80% = slow queries</li><li>Connections near max = pool exhaustion</li><li>Sudden spikes = query or app issue</li></ul><p><strong>6. Application Logs</strong></p><ul><li><strong>Shows:</strong> Recent app logs.</li><li><strong>Why:</strong> Immediate visibility into errors and trends.</li><li><strong>Watch for:</strong> Error bursts, stack traces, unusual log volume.</li></ul><h3>Implementation</h3><p>Create terraform/cloudwatch-dashboard.tf:</p><pre>resource &quot;aws_cloudwatch_dashboard&quot; &quot;main&quot; {<br>  dashboard_name = &quot;${var.project_name}-${var.environment}-dashboard&quot;<br><br>  dashboard_body = jsonencode({<br>    widgets = [<br>      # ECS Core Health (CPU &amp; Memory)<br>      {<br>        type   = &quot;metric&quot;<br>        x      = 0<br>        y      = 0<br>        width  = 12<br>        height = 6<br>        properties = {<br>          metrics = [<br>            [&quot;AWS/ECS&quot;, &quot;CPUUtilization&quot;, &quot;ServiceName&quot;, aws_ecs_service.main.name, &quot;ClusterName&quot;, aws_ecs_cluster.main.name, { stat = &quot;Average&quot;, label = &quot;CPU %&quot;, color = &quot;#1f77b4&quot; }],<br>            [&quot;.&quot;, &quot;MemoryUtilization&quot;, &quot;.&quot;, &quot;.&quot;, &quot;.&quot;, &quot;.&quot;, { stat = &quot;Average&quot;, label = &quot;Memory %&quot;, color = &quot;#ff7f0e&quot; }]<br>          ]<br>          period = 300<br>          region = var.aws_region<br>          title  = &quot;ECS Health: CPU/Memory/Tasks&quot;<br>          yAxis = { left = { min = 0, max = 100, label = &quot;%&quot; } }<br>        }<br>      },<br><br>      # ALB Request Rate &amp; Response Time<br>      {<br>        type   = &quot;metric&quot;<br>        x      = 12<br>        y      = 0<br>        width  = 12<br>        height = 6<br>        properties = {<br>          metrics = [<br>            # Requests (Left Axis)<br>            [&quot;AWS/ApplicationELB&quot;, &quot;RequestCount&quot;, &quot;LoadBalancer&quot;, aws_lb.main.arn_suffix, { stat = &quot;Sum&quot;, label = &quot;Total Requests&quot;, yAxis = &quot;left&quot; }],<br>            # Response Time (Right Axis)<br>            [&quot;.&quot;, &quot;TargetResponseTime&quot;, &quot;.&quot;, &quot;.&quot;, { stat = &quot;p90&quot;, label = &quot;p90 Response Time (s)&quot;, yAxis = &quot;right&quot;, color = &quot;#d62728&quot; }]<br>          ]<br>          period = 300<br>          region = var.aws_region<br>          title  = &quot;ALB: Requests &amp; Latency (P90)&quot;<br>          yAxis = {<br>            left  = { label = &quot;Requests&quot; }<br>            right = { label = &quot;Seconds&quot; }<br>          }<br>        }<br>      },<br><br>      # ALB Error Rates (5XX) &amp; Target Health<br>      {<br>        type   = &quot;metric&quot;<br>        x      = 0<br>        y      = 6<br>        width  = 12<br>        height = 6<br>        properties = {<br>          metrics = [<br>            [&quot;AWS/ApplicationELB&quot;, &quot;HTTPCode_Target_5XX_Count&quot;, &quot;LoadBalancer&quot;, aws_lb.main.arn_suffix, { stat = &quot;Sum&quot;, label = &quot;5XX Server Error&quot;, color = &quot;#d62728&quot; }],<br>            [&quot;.&quot;, &quot;UnHealthyHostCount&quot;, &quot;TargetGroup&quot;, aws_lb_target_group.main.arn_suffix, &quot;LoadBalancer&quot;, aws_lb.main.arn_suffix, { stat = &quot;Average&quot;, label = &quot;Unhealthy Targets&quot;, color = &quot;#ff7f0e&quot; }],<br>            [&quot;.&quot;, &quot;HealthyHostCount&quot;, &quot;.&quot;, &quot;.&quot;, &quot;.&quot;, &quot;.&quot;, { stat = &quot;Average&quot;, label = &quot;Healthy Targets&quot;, color = &quot;#2ca02c&quot; }]<br>          ]<br>          period = 60<br>          region = var.aws_region<br>          title  = &quot;ALB/Target Health &amp; 5XX Errors&quot;<br>          yAxis = { left = { min = 0 } }<br>        }<br>      },<br><br>      # RDS CPU and Connections<br>      {<br>        type   = &quot;metric&quot;<br>        x      = 12<br>        y      = 6<br>        width  = 12<br>        height = 6<br>        properties = {<br>          metrics = [<br>            [&quot;AWS/RDS&quot;, &quot;CPUUtilization&quot;, &quot;DBInstanceIdentifier&quot;, &quot;${var.project_name}-${var.environment}-db&quot;, { stat = &quot;Average&quot;, label = &quot;DB CPU %&quot;, yAxis = &quot;left&quot;, color = &quot;#1f77b4&quot; }],<br>            [&quot;.&quot;, &quot;DatabaseConnections&quot;, &quot;.&quot;, &quot;.&quot;, { stat = &quot;Average&quot;, label = &quot;DB Connections&quot;, yAxis = &quot;right&quot;, color = &quot;#2ca02c&quot; }]<br>          ]<br>          period = 300<br>          region = var.aws_region<br>          title  = &quot;RDS Core Metrics (CPU &amp; Connections)&quot;<br>          yAxis = { left = { min = 0, max = 100, label = &quot;%&quot; }, right = { min = 0, label = &quot;Count&quot; } }<br>        }<br>      },<br><br>      # Recent Application Logs<br>      {<br>        type   = &quot;log&quot;<br>        x      = 0<br>        y      = 12<br>        width  = 24<br>        height = 8<br>        properties = {<br>          query  = &quot;SOURCE &#39;${aws_cloudwatch_log_group.ecs.name}&#39; | fields @timestamp, @message | sort @timestamp desc | limit 100&quot;<br>          region = var.aws_region<br>          title  = &quot;Recent Application Logs&quot;<br>        }<br>      }<br>    ]<br>  })<br>}<br><br>output &quot;cloudwatch_dashboard_url&quot; {<br>  description = &quot;URL to view the CloudWatch Dashboard&quot;<br>  value       = &quot;https://console.aws.amazon.com/cloudwatch/home?region=${var.aws_region}#dashboards:name=${aws_cloudwatch_dashboard.main.dashboard_name}&quot;<br>}</pre><h3>Deploy the Dashboard</h3><pre># cd into the project directory<br><br>git add .<br>git commit -m &#39;cloudwatch-dashboard&#39;<br>git push origin</pre><p>On the AWS console, navigate to Cloudwatch &gt; Dashboards and you would see your application dashboard</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5Hlx4oQ7t__9mXPGnBL2A.png" /></figure><h3>Pro Tips</h3><p>💡 <strong>Bookmark your dashboard</strong> — You’ll visit it multiple times daily</p><p>💡 <strong>Set it as homepage</strong> — Some teams display it on office screens</p><p>💡 <strong>Check before deploying</strong> — Verify baseline before releases</p><p>💡 <strong>Check after deploying</strong> — Spot issues immediately</p><p>💡 <strong>Use in incident calls</strong> — Share screen to show team current state</p><h3>Component 2: CloudWatch Alarms</h3><h3>CloudWatch Alarms</h3><p><strong>What They Are:</strong><br> Automated watchers that monitor metrics and alert you when issues arise.</p><p><strong>Why You Need Them:</strong><br> Dashboards show problems — but only if someone’s watching.<br> CloudWatch Alarms work 24/7, instantly notifying you when thresholds are breached, even at 3 AM.</p><h3>The Critical 5 Alarms</h3><p><strong>1. ECS High CPU (80%+ for 10 min)</strong></p><ul><li><strong>Why it matters:</strong> Early sign of overload or bad code; prevents crashes.</li><li><strong>Causes:</strong> Traffic spike, inefficient code, memory thrashing, under-sized tasks.</li><li><strong>Action:</strong> Check traffic and deployments, scale up, debug code if persistent.</li></ul><p><strong>2. ECS High Memory (80%+ for 10 min)</strong></p><ul><li><strong>Why it matters:</strong> Avoids OOM kills and task restarts.</li><li><strong>Causes:</strong> Memory leaks, large in-memory data, low task memory, big file loads.</li><li><strong>Action:</strong> Identify leak vs. spike, inspect heap, restart temporarily, fix leak.</li></ul><p><strong>3. Unhealthy Targets (ALB)</strong></p><ul><li><strong>Why it matters:</strong> ALB stops routing to unhealthy targets → downtime risk.</li><li><strong>Causes:</strong> App crash, broken health check, network/security issue, deployment.</li><li><strong>Action:</strong> Check ECS task and logs, verify health endpoint, review SG rules.</li></ul><p><strong>4. High 5XX Errors (10+ in 5 min)</strong></p><ul><li><strong>Why it matters:</strong> Users hit server errors; often recent deployment issue.</li><li><strong>Causes:</strong> Bad release, DB errors, unhandled exceptions, dependency outage.</li><li><strong>Action:</strong> Inspect logs, review last deploy, rollback or patch, redeploy fix.</li></ul><p><strong>5. RDS High CPU (80%+ for 10 min)</strong></p><ul><li><strong>Why it matters:</strong> DB bottleneck slows entire stack.</li><li><strong>Causes:</strong> Inefficient queries, missing indexes, table scans, excess connections.</li><li><strong>Action:</strong> Check slow query log, tune queries, add indexes, scale DB if needed.</li></ul><h3>Understanding Alarm Configuration</h3><p>Before we implement, let’s understand the key parameters:</p><pre>threshold           = 80        # Value that triggers alarm<br>evaluation_periods  = 2         # How many periods must breach<br>period              = 300       # Time window (5 minutes)<br>statistic          = &quot;Average&quot;  # How to aggregate data<br>comparison_operator = &quot;GreaterThanThreshold&quot;<br>```<br>**Example: ECS High CPU Alarm**<br>```<br>threshold = 80<br>evaluation_periods = 2<br>period = 300<br>Means: Alert if CPU &gt; 80% for 2 consecutive 5-minute periods (10 minutes total)</pre><p><strong>Why 2 evaluation periods?</strong></p><ul><li>1 period = Too sensitive, false alarms on brief spikes</li><li>2 periods = Balances responsiveness with reliability</li><li>3+ periods = Might miss critical issues</li></ul><h3>Implementation</h3><p>Create terraform/cloudwatch-alarms.tf:</p><pre># ============================================<br># Core Infrastructure Health Alarms (ECS, ALB, RDS)<br># GitOps status/drift alarms removed due to external workflows<br># ============================================<br><br># 1. Slow Deployments<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;slow_deployments&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-slow-deployments&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 1<br>  metric_name         = &quot;DeploymentDuration&quot;<br>  namespace           = &quot;AttendanceApp/Deployments&quot;<br>  period              = 3600<br>  statistic           = &quot;Average&quot;<br>  threshold           = 600 # 10 minutes<br>  alarm_description   = &quot;Alert when deployments take longer than 10 minutes&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  treat_missing_data  = &quot;notBreaching&quot;<br>  dimensions          = { Environment = var.environment }<br>}<br><br># 2. ECS High CPU<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;ecs_high_cpu&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-ecs-high-cpu&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 2<br>  metric_name         = &quot;CPUUtilization&quot;<br>  namespace           = &quot;AWS/ECS&quot;<br>  period              = 300<br>  statistic           = &quot;Average&quot;<br>  threshold           = 80<br>  alarm_description   = &quot;ECS CPU usage is above 80%&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions = {<br>    ServiceName = aws_ecs_service.main.name<br>    ClusterName = aws_ecs_cluster.main.name<br>  }<br>}<br><br># 3. ECS High Memory<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;ecs_high_memory&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-ecs-high-memory&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 2<br>  metric_name         = &quot;MemoryUtilization&quot;<br>  namespace           = &quot;AWS/ECS&quot;<br>  period              = 300<br>  statistic           = &quot;Average&quot;<br>  threshold           = 80<br>  alarm_description   = &quot;ECS Memory usage is above 80%&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions = {<br>    ServiceName = aws_ecs_service.main.name<br>    ClusterName = aws_ecs_cluster.main.name<br>  }<br>}<br><br># 4. ECS No Running Tasks<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;ecs_no_tasks&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-ecs-no-tasks&quot;<br>  comparison_operator = &quot;LessThanThreshold&quot;<br>  evaluation_periods  = 1<br>  metric_name         = &quot;RunningTasksCount&quot;<br>  namespace           = &quot;AWS/ECS&quot;<br>  period              = 60<br>  statistic           = &quot;Average&quot;<br>  threshold           = 1<br>  alarm_description   = &quot;No ECS tasks are running&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions = {<br>    ServiceName = aws_ecs_service.main.name<br>    ClusterName = aws_ecs_cluster.main.name<br>  }<br>}<br><br># 5. ALB Unhealthy Targets<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;unhealthy_targets&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-unhealthy-targets&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 1<br>  metric_name         = &quot;UnHealthyHostCount&quot;<br>  namespace           = &quot;AWS/ApplicationELB&quot;<br>  period              = 60<br>  statistic           = &quot;Average&quot;<br>  threshold           = 0<br>  alarm_description   = &quot;ALB has unhealthy targets&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions = {<br>    TargetGroup  = aws_lb_target_group.main.arn_suffix<br>    LoadBalancer = aws_lb.main.arn_suffix<br>  }<br>}<br><br># 6. ALB High Response Time<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;high_response_time&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-high-response-time&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 2<br>  metric_name         = &quot;TargetResponseTime&quot;<br>  namespace           = &quot;AWS/ApplicationELB&quot;<br>  period              = 300<br>  statistic           = &quot;Average&quot;<br>  threshold           = 2 # 2 seconds<br>  alarm_description   = &quot;ALB response time is above 2 seconds&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions          = { LoadBalancer = aws_lb.main.arn_suffix }<br>}<br><br># 7. ALB High 5XX Errors<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;high_5xx_errors&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-high-5xx-errors&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 1<br>  metric_name         = &quot;HTTPCode_Target_5XX_Count&quot;<br>  namespace           = &quot;AWS/ApplicationELB&quot;<br>  period              = 300<br>  statistic           = &quot;Sum&quot;<br>  threshold           = 10<br>  alarm_description   = &quot;More than 10 5XX errors in 5 minutes&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  treat_missing_data  = &quot;notBreaching&quot;<br>  dimensions          = { LoadBalancer = aws_lb.main.arn_suffix }<br>}<br><br># 8. ALB High 4XX Errors<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;high_4xx_errors&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-high-4xx-errors&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 1<br>  metric_name         = &quot;HTTPCode_Target_4XX_Count&quot;<br>  namespace           = &quot;AWS/ApplicationELB&quot;<br>  period              = 300<br>  statistic           = &quot;Sum&quot;<br>  threshold           = 50 # More lenient for 4XX<br>  alarm_description   = &quot;More than 50 4XX errors in 5 minutes&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  treat_missing_data  = &quot;notBreaching&quot;<br>  dimensions          = { LoadBalancer = aws_lb.main.arn_suffix }<br>}<br><br># 9. RDS High CPU<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;rds_high_cpu&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-rds-high-cpu&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 2<br>  metric_name         = &quot;CPUUtilization&quot;<br>  namespace           = &quot;AWS/RDS&quot;<br>  period              = 300<br>  statistic           = &quot;Average&quot;<br>  threshold           = 80<br>  alarm_description   = &quot;RDS CPU usage is above 80%&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions          = { DBInstanceIdentifier = &quot;${var.project_name}-${var.environment}-db&quot; }<br>}<br><br># 10. RDS Low Storage<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;rds_low_storage&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-rds-low-storage&quot;<br>  comparison_operator = &quot;LessThanThreshold&quot;<br>  evaluation_periods  = 1<br>  metric_name         = &quot;FreeStorageSpace&quot;<br>  namespace           = &quot;AWS/RDS&quot;<br>  period              = 300<br>  statistic           = &quot;Average&quot;<br>  threshold           = 5000000000 # 5GB<br>  alarm_description   = &quot;RDS has less than 5GB free storage&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions          = { DBInstanceIdentifier = &quot;${var.project_name}-${var.environment}-db&quot; }<br>}<br><br># 11. RDS High Connections<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;rds_high_connections&quot; {<br>  alarm_name          = &quot;${var.project_name}-${var.environment}-rds-high-connections&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 2<br>  metric_name         = &quot;DatabaseConnections&quot;<br>  namespace           = &quot;AWS/RDS&quot;<br>  period              = 300<br>  statistic           = &quot;Average&quot;<br>  threshold           = 80 # Adjust based on your max_connections<br>  alarm_description   = &quot;RDS has more than 80 connections&quot;<br>  alarm_actions       = [aws_sns_topic.alerts.arn]<br>  dimensions          = { DBInstanceIdentifier = &quot;${var.project_name}-${var.environment}-db&quot; }<br>}<br><br># ============================================<br># SNS Topic and Subscriptions<br># ============================================<br><br>resource &quot;aws_sns_topic&quot; &quot;alerts&quot; {<br>  name = &quot;${var.project_name}-${var.environment}-alerts&quot;<br>  tags = {<br>    Name        = &quot;${var.project_name}-${var.environment}-alerts&quot;<br>    Environment = var.environment<br>  }<br>}<br><br>resource &quot;aws_sns_topic_subscription&quot; &quot;email_alerts&quot; {<br>  topic_arn = aws_sns_topic.alerts.arn<br>  protocol  = &quot;email&quot;<br>  endpoint  = var.alert_email<br>}<br><br># Outputs<br>output &quot;sns_topic_arn&quot; {<br>  description = &quot;SNS Topic ARN for alerts&quot;<br>  value       = aws_sns_topic.alerts.arn<br>}<br><br>output &quot;alarm_names&quot; {<br>  description = &quot;List of all CloudWatch alarm names&quot;<br>  value = [<br>    aws_cloudwatch_metric_alarm.slow_deployments.alarm_name,<br>    aws_cloudwatch_metric_alarm.ecs_high_cpu.alarm_name,<br>    aws_cloudwatch_metric_alarm.ecs_high_memory.alarm_name,<br>    aws_cloudwatch_metric_alarm.ecs_no_tasks.alarm_name,<br>    aws_cloudwatch_metric_alarm.unhealthy_targets.alarm_name,<br>    aws_cloudwatch_metric_alarm.high_response_time.alarm_name,<br>    aws_cloudwatch_metric_alarm.high_5xx_errors.alarm_name,<br>    aws_cloudwatch_metric_alarm.high_4xx_errors.alarm_name,<br>    aws_cloudwatch_metric_alarm.rds_high_cpu.alarm_name,<br>    aws_cloudwatch_metric_alarm.rds_low_storage.alarm_name,<br>    aws_cloudwatch_metric_alarm.rds_high_connections.alarm_name,<br>  ]<br>}</pre><h3>Add Variable</h3><p>In terraform/variables.tf:</p><pre>variable &quot;alert_email&quot; {<br>  description = &quot;Email address for CloudWatch alerts&quot;<br>  type        = string<br>  default     = &quot;tobi.akinlade@outlook.com&quot;<br>}</pre><h3>Deploy the Alarms</h3><pre>cd project directory<br><br>git add .<br>git commit -m &#39;cloudwatch-alarms&#39;<br>git push origin main</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAXEYPNNHOuDePFJRJ5O9g.png" /></figure><h3>Testing the Alarms</h3><p>From the terminal, cd in to the project directory and create a file for the test script.</p><pre># create file<br><br>test-cloud-alarm.sh<br><br># Paste this into the file <br><br>#!/bin/bash<br><br># ============================================<br># CloudWatch Alarm Test Script<br># ============================================<br># This script triggers your CloudWatch alarms for testing<br># No dedicated Linux machine needed - run from anywhere!<br>#<br># Prerequisites:<br># - AWS CLI installed and configured<br># - jq installed (for JSON parsing)<br># - Appropriate AWS permissions<br># ============================================<br><br>set -e  # Exit on error<br><br># Configuration<br>REGION=&quot;eu-west-2&quot;<br>ENVIRONMENT=&quot;dev&quot;<br>PROJECT=&quot;attendance-app&quot;<br>CLUSTER=&quot;${PROJECT}-${ENVIRONMENT}-cluster&quot;<br>SERVICE=&quot;${PROJECT}-${ENVIRONMENT}-service&quot;<br>DB_INSTANCE=&quot;${PROJECT}-${ENVIRONMENT}-db&quot;<br><br># Colors for output<br>RED=&#39;\033[0;31m&#39;<br>GREEN=&#39;\033[0;32m&#39;<br>YELLOW=&#39;\033[1;33m&#39;<br>BLUE=&#39;\033[0;34m&#39;<br>NC=&#39;\033[0m&#39; # No Color<br><br># ============================================<br># Helper Functions<br># ============================================<br><br>print_header() {<br>    echo &quot;&quot;<br>    echo -e &quot;${BLUE}============================================${NC}&quot;<br>    echo -e &quot;${BLUE}$1${NC}&quot;<br>    echo -e &quot;${BLUE}============================================${NC}&quot;<br>    echo &quot;&quot;<br>}<br><br>print_success() {<br>    echo -e &quot;${GREEN}✅ $1${NC}&quot;<br>}<br><br>print_warning() {<br>    echo -e &quot;${YELLOW}⚠️  $1${NC}&quot;<br>}<br><br>print_error() {<br>    echo -e &quot;${RED}❌ $1${NC}&quot;<br>}<br><br>wait_for_alarm() {<br>    local alarm_name=$1<br>    local max_wait=300  # 5 minutes<br>    local waited=0<br>    <br>    echo &quot;⏳ Waiting for alarm to trigger (max 5 minutes)...&quot;<br>    <br>    while [ $waited -lt $max_wait ]; do<br>        STATE=$(aws cloudwatch describe-alarms \<br>            --alarm-names &quot;$alarm_name&quot; \<br>            --region $REGION \<br>            --query &#39;MetricAlarms[0].StateValue&#39; \<br>            --output text)<br>        <br>        if [ &quot;$STATE&quot; == &quot;ALARM&quot; ]; then<br>            print_success &quot;Alarm triggered! Current state: $STATE&quot;<br>            return 0<br>        fi<br>        <br>        echo &quot;   Current state: $STATE (waited ${waited}s)&quot;<br>        sleep 15<br>        waited=$((waited + 15))<br>    done<br>    <br>    print_warning &quot;Alarm did not trigger within $max_wait seconds&quot;<br>    return 1<br>}<br><br># ============================================<br># Test 1: Deployment Failure Alarm<br># ============================================<br><br>test_deployment_failure() {<br>    print_header &quot;TEST 1: Triggering Deployment Failure Alarm&quot;<br>    <br>    ALARM_NAME=&quot;${PROJECT}-${ENVIRONMENT}-deployment-failures&quot;<br>    <br>    print_warning &quot;Sending fake deployment failure metric...&quot;<br>    <br>    aws cloudwatch put-metric-data \<br>        --namespace &quot;AttendanceApp/Deployments&quot; \<br>        --metric-name &quot;DeploymentCount&quot; \<br>        --value 1 \<br>        --dimensions Environment=$ENVIRONMENT,Status=failure \<br>        --timestamp $(date -u +%Y-%m-%dT%H:%M:%SZ) \<br>        --region $REGION<br>    <br>    print_success &quot;Metric sent!&quot;<br>    wait_for_alarm &quot;$ALARM_NAME&quot;<br>}<br><br># ============================================<br># Test 2: Infrastructure Drift Alarm<br># ============================================<br><br>test_drift_detection() {<br>    print_header &quot;TEST 2: Triggering Drift Detection Alarm&quot;<br>    <br>    ALARM_NAME=&quot;${PROJECT}-${ENVIRONMENT}-drift-detected&quot;<br>    <br>    print_warning &quot;Sending fake drift detection metric...&quot;<br>    <br>    aws cloudwatch put-metric-data \<br>        --namespace &quot;AttendanceApp/Drift&quot; \<br>        --metric-name &quot;InfrastructureDrift&quot; \<br>        --value 1 \<br>        --dimensions Environment=$ENVIRONMENT,Type=terraform,Status=detected \<br>        --timestamp $(date -u +%Y-%m-%dT%H:%M:%SZ) \<br>        --region $REGION<br>    <br>    print_success &quot;Metric sent!&quot;<br>    wait_for_alarm &quot;$ALARM_NAME&quot;<br>}<br><br># ============================================<br># Test 3: Slow Deployment Alarm<br># ============================================<br><br>test_slow_deployment() {<br>    print_header &quot;TEST 3: Triggering Slow Deployment Alarm&quot;<br>    <br>    ALARM_NAME=&quot;${PROJECT}-${ENVIRONMENT}-slow-deployments&quot;<br>    <br>    print_warning &quot;Sending slow deployment metric (15 minutes)...&quot;<br>    <br>    aws cloudwatch put-metric-data \<br>        --namespace &quot;AttendanceApp/Deployments&quot; \<br>        --metric-name &quot;DeploymentDuration&quot; \<br>        --value 900 \<br>        --unit Seconds \<br>        --dimensions Environment=$ENVIRONMENT \<br>        --timestamp $(date -u +%Y-%m-%dT%H:%M:%SZ) \<br>        --region $REGION<br>    <br>    print_success &quot;Metric sent!&quot;<br>    wait_for_alarm &quot;$ALARM_NAME&quot;<br>}<br><br># ============================================<br># Test 4: ALB 5XX Errors<br># ============================================<br><br>test_5xx_errors() {<br>    print_header &quot;TEST 4: Triggering High 5XX Errors Alarm&quot;<br>    <br>    ALARM_NAME=&quot;${PROJECT}-${ENVIRONMENT}-high-5xx-errors&quot;<br>    <br>    # Get ALB DNS name<br>    ALB_DNS=$(aws elbv2 describe-load-balancers \<br>        --names &quot;${PROJECT}-${ENVIRONMENT}-alb&quot; \<br>        --region $REGION \<br>        --query &#39;LoadBalancers[0].DNSName&#39; \<br>        --output text 2&gt;/dev/null || echo &quot;&quot;)<br>    <br>    if [ -z &quot;$ALB_DNS&quot; ]; then<br>        print_error &quot;Could not find ALB. Skipping test.&quot;<br>        return 1<br>    fi<br>    <br>    print_warning &quot;Generating 5XX errors by hitting non-existent endpoints...&quot;<br>    <br>    # Hit non-existent endpoints to generate 503 errors<br>    for i in {1..15}; do<br>        curl -s -o /dev/null -w &quot;%{http_code}\n&quot; \<br>            &quot;http://${ALB_DNS}/trigger-error-$i&quot; || true<br>        echo &quot;   Request $i sent&quot;<br>        sleep 2<br>    done<br>    <br>    print_success &quot;Requests sent!&quot;<br>    wait_for_alarm &quot;$ALARM_NAME&quot;<br>}<br><br># ============================================<br># Test 5: Generate Load on Application<br># ============================================<br><br>test_generate_load() {<br>    print_header &quot;TEST 5: Generating Application Load&quot;<br>    <br>    # Get application URL<br>    APP_URL=$(aws elbv2 describe-load-balancers \<br>        --names &quot;${PROJECT}-${ENVIRONMENT}-alb&quot; \<br>        --region $REGION \<br>        --query &#39;LoadBalancers[0].DNSName&#39; \<br>        --output text 2&gt;/dev/null || echo &quot;&quot;)<br>    <br>    if [ -z &quot;$APP_URL&quot; ]; then<br>        print_error &quot;Could not find application URL. Skipping test.&quot;<br>        return 1<br>    fi<br>    <br>    print_warning &quot;Generating load to potentially trigger CPU/Memory alarms...&quot;<br>    print_warning &quot;Sending 100 concurrent requests...&quot;<br>    <br>    # Generate load<br>    for i in {1..100}; do<br>        curl -s &quot;http://${APP_URL}/&quot; &gt; /dev/null &amp;<br>        curl -s &quot;http://${APP_URL}/api/health&quot; &gt; /dev/null &amp;<br>    done<br>    <br>    wait  # Wait for all background jobs<br>    <br>    print_success &quot;Load generated!&quot;<br>    print_warning &quot;Monitor ECS CPU/Memory alarms for 5-10 minutes&quot;<br>}<br><br># ============================================<br># Test 6: Simulate Database Load<br># ============================================<br><br>test_database_load() {<br>    print_header &quot;TEST 6: Database Load (Simulated)&quot;<br>    <br>    print_warning &quot;To test RDS alarms, you need to:&quot;<br>    echo &quot;   1. Connect to your RDS instance&quot;<br>    echo &quot;   2. Run queries to generate CPU load&quot;<br>    echo &quot;   3. Open multiple connections&quot;<br>    echo &quot;&quot;<br>    echo &quot;Example connection command:&quot;<br>    echo &quot;   psql -h ${DB_INSTANCE}.xxxxx.${REGION}.rds.amazonaws.com -U your_user -d your_db&quot;<br>    echo &quot;&quot;<br>    echo &quot;Example load query:&quot;<br>    echo &quot;   SELECT pg_sleep(1) FROM generate_series(1, 1000);&quot;<br>    echo &quot;&quot;<br>    print_warning &quot;Skipping automated RDS load test (requires DB credentials)&quot;<br>}<br><br># ============================================<br># Test 7: Check All Alarm States<br># ============================================<br><br>check_all_alarms() {<br>    print_header &quot;Checking All Alarm States&quot;<br>    <br>    aws cloudwatch describe-alarms \<br>        --alarm-name-prefix &quot;${PROJECT}-${ENVIRONMENT}&quot; \<br>        --region $REGION \<br>        --query &#39;MetricAlarms[*].[AlarmName,StateValue]&#39; \<br>        --output table<br>}<br><br># ============================================<br># Main Menu<br># ============================================<br><br>show_menu() {<br>    echo &quot;&quot;<br>    echo &quot;======================================&quot;<br>    echo &quot;  CloudWatch Alarm Test Menu&quot;<br>    echo &quot;======================================&quot;<br>    echo &quot;&quot;<br>    echo &quot;  1) Test Deployment Failure Alarm&quot;<br>    echo &quot;  2) Test Drift Detection Alarm&quot;<br>    echo &quot;  3) Test Slow Deployment Alarm&quot;<br>    echo &quot;  4) Test High 5XX Errors Alarm&quot;<br>    echo &quot;  5) Generate Application Load&quot;<br>    echo &quot;  6) Database Load Info&quot;<br>    echo &quot;  7) Check All Alarm States&quot;<br>    echo &quot;  8) Run All Tests&quot;<br>    echo &quot;  9) Exit&quot;<br>    echo &quot;&quot;<br>    echo -n &quot;Select option [1-9]: &quot;<br>}<br><br>run_all_tests() {<br>    print_header &quot;RUNNING ALL TESTS&quot;<br>    <br>    test_deployment_failure<br>    sleep 5<br>    <br>    test_drift_detection<br>    sleep 5<br>    <br>    test_slow_deployment<br>    sleep 5<br>    <br>    test_5xx_errors<br>    sleep 5<br>    <br>    test_generate_load<br>    <br>    print_header &quot;ALL TESTS COMPLETE&quot;<br>    <br>    echo &quot;&quot;<br>    print_warning &quot;Checking alarm states in 2 minutes...&quot;<br>    sleep 120<br>    <br>    check_all_alarms<br>}<br><br># ============================================<br># Main Script<br># ============================================<br><br>main() {<br>    print_header &quot;CloudWatch Alarm Test Script&quot;<br>    <br>    echo &quot;Configuration:&quot;<br>    echo &quot;  Region:      $REGION&quot;<br>    echo &quot;  Environment: $ENVIRONMENT&quot;<br>    echo &quot;  Cluster:     $CLUSTER&quot;<br>    echo &quot;  Service:     $SERVICE&quot;<br>    echo &quot;&quot;<br>    <br>    # Check prerequisites<br>    if ! command -v aws &amp;&gt; /dev/null; then<br>        print_error &quot;AWS CLI not found. Please install it first.&quot;<br>        exit 1<br>    fi<br>    <br>    if ! command -v jq &amp;&gt; /dev/null; then<br>        print_warning &quot;jq not found. Some features may not work.&quot;<br>    fi<br>    <br>    # Interactive menu<br>    while true; do<br>        show_menu<br>        read -r choice<br>        <br>        case $choice in<br>            1) test_deployment_failure ;;<br>            2) test_drift_detection ;;<br>            3) test_slow_deployment ;;<br>            4) test_5xx_errors ;;<br>            5) test_generate_load ;;<br>            6) test_database_load ;;<br>            7) check_all_alarms ;;<br>            8) run_all_tests ;;<br>            9) <br>                print_success &quot;Goodbye!&quot;<br>                exit 0<br>                ;;<br>            *)<br>                print_error &quot;Invalid option&quot;<br>                ;;<br>        esac<br>        <br>        echo &quot;&quot;<br>        echo &quot;Press Enter to continue...&quot;<br>        read -r<br>    done<br>}<br><br># Run main function<br>main</pre><p>Make the file executable and execute</p><pre>chmod +x test-cloud-alarm.sh<br><br>./test-cloud-alarm.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oxZ0HEgRTFtuTeRjJ1r6xg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMc1XdazSQKq7eRjA0IXUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-kvWFfn3zQWXsv1Vd9zDw.png" /></figure><h3>Component 3: Drift Detection</h3><p>This is the key to drift detection:</p><pre>terraform plan -detailed-exitcode<br>Exit Code 0: Success, no changes needed (no drift)<br>Exit Code 1: Error occurred (configuration issue)<br>Exit Code 2: Success, changes detected (DRIFT!)</pre><p><strong>Why this matters:</strong></p><p>Most people think any non-zero exit code is an error. But <strong>exit code 2 is success</strong> — it successfully detected changes!</p><h3>Implementation</h3><p>Create .github/workflows/terraform-drift-detection.yml:</p><pre>name: Terraform Drift Detection<br><br>on:<br>  schedule:<br>    # Runs every hour, offset by 10 minutes (at 00:10, 01:10, 02:10, etc.)<br>    - cron: &#39;10 * * * *&#39;<br>  workflow_dispatch:<br><br>env:<br>  TF_VERSION: &#39;1.6.0&#39;<br>  AWS_REGION: eu-west-2<br><br>jobs:<br>  detect-drift:<br>    name: Detect Infrastructure Drift<br>    runs-on: ubuntu-latest<br>    permissions:<br>      id-token: write<br>      contents: read<br>      issues: write<br><br>    defaults:<br>      run:<br>        working-directory: ./terraform<br><br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br>          aws-region: ${{ env.AWS_REGION }}<br><br>      - name: Setup Terraform<br>        uses: hashicorp/setup-terraform@v3<br>        with:<br>          terraform_version: ${{ env.TF_VERSION }}<br>          terraform_wrapper: false<br><br>      - name: Terraform Init<br>        run: |<br>          terraform init \<br>            -backend-config=&quot;bucket=${{ secrets.TF_STATE_BUCKET }}&quot; \<br>            -backend-config=&quot;key=attendance-app/terraform.tfstate&quot; \<br>            -backend-config=&quot;region=${{ env.AWS_REGION }}&quot;<br><br>      - name: Terraform Plan (Detect Drift)<br>        id: plan<br>        run: |<br>          set +e<br>          terraform plan -input=false -detailed-exitcode -no-color \<br>            -var=&quot;environment=dev&quot; \<br>            -var=&quot;project_name=attendance-app&quot; \<br>            -var=&quot;domain_name=tech-with-tobi.com&quot; \<br>            -var=&quot;route53_zone_name=tech-with-tobi.com&quot; \<br>            -var=&quot;create_route53_record=true&quot; \<br>            -var=&quot;desired_count=2&quot; \<br>            -var=&quot;db_username=${{ secrets.DB_USERNAME }}&quot; \<br>            -var=&quot;db_password=${{ secrets.DB_PASSWORD }}&quot; \<br>            -out=tfplan 2&gt;&amp;1 | tee plan_output.txt<br>          EXIT_CODE=${PIPESTATUS[0]}<br><br>          echo &quot;&quot;<br>          echo &quot;📊 Terraform plan completed&quot;<br>          echo &quot;   Exit code: $EXIT_CODE&quot;<br>          echo &quot;   0 = No changes (no drift)&quot;<br>          echo &quot;   1 = Error occurred&quot;<br>          echo &quot;   2 = Changes detected (DRIFT!)&quot;<br><br>          echo &quot;exit_code=$EXIT_CODE&quot; &gt;&gt; $GITHUB_OUTPUT<br>          exit 0<br><br>      - name: Check for Drift<br>        id: drift<br>        run: |<br>          EXIT_CODE=&quot;${{ steps.plan.outputs.exit_code }}&quot;<br>          echo &quot;Checking exit code: $EXIT_CODE&quot;<br>          <br>          if [ &quot;$EXIT_CODE&quot; == &quot;2&quot; ]; then<br>            echo &quot;drift_detected=true&quot; &gt;&gt; $GITHUB_OUTPUT<br>            echo &quot;🚨 Drift detected!&quot;<br>          else<br>            echo &quot;drift_detected=false&quot; &gt;&gt; $GITHUB_OUTPUT<br>            echo &quot;✅ No drift detected&quot;<br>          fi<br><br>      - name: Record Drift Detection in CloudWatch<br>        if: steps.drift.outputs.drift_detected == &#39;true&#39;<br>        run: |<br>          TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)<br>          <br>          echo &quot;📊 Recording drift detection in CloudWatch...&quot;<br>          <br>          # 1. Record drift metric<br>          aws cloudwatch put-metric-data \<br>            --namespace &quot;AttendanceApp/Drift&quot; \<br>            --metric-name &quot;InfrastructureDrift&quot; \<br>            --value 1 \<br>            --dimensions Environment=dev,Type=terraform,Status=detected \<br>            --timestamp $TIMESTAMP \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          # 2. Record drift detection event<br>          aws cloudwatch put-metric-data \<br>            --namespace &quot;AttendanceApp/Events&quot; \<br>            --metric-name &quot;DriftDetectionEvent&quot; \<br>            --value 1 \<br>            --dimensions Environment=dev,Severity=high \<br>            --timestamp $TIMESTAMP \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          # 3. Create CloudWatch Logs group if it doesn&#39;t exist<br>          aws logs create-log-group \<br>            --log-group-name /attendanceapp/drift-detection \<br>            --region ${{ env.AWS_REGION }} 2&gt;/dev/null || true<br>          <br>          # 4. Create log stream for today<br>          aws logs create-log-stream \<br>            --log-group-name /attendanceapp/drift-detection \<br>            --log-stream-name &quot;drift-$(date +%Y%m%d)&quot; \<br>            --region ${{ env.AWS_REGION }} 2&gt;/dev/null || true<br>          <br>          # 5. Log drift details to CloudWatch Logs<br>          MESSAGE=$(cat &lt;&lt; EOF<br>          {<br>            &quot;timestamp&quot;: &quot;$TIMESTAMP&quot;,<br>            &quot;event&quot;: &quot;drift_detected&quot;,<br>            &quot;environment&quot;: &quot;dev&quot;,<br>            &quot;terraform_exit_code&quot;: &quot;${{ steps.plan.outputs.exit_code }}&quot;,<br>            &quot;workflow_run&quot;: &quot;https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}&quot;,<br>            &quot;triggered_by&quot;: &quot;${{ github.actor }}&quot;,<br>            &quot;drift_details&quot;: &quot;Infrastructure has diverged from Terraform state&quot;<br>          }<br>          EOF<br>          )<br>          <br>          aws logs put-log-events \<br>            --log-group-name /attendanceapp/drift-detection \<br>            --log-stream-name &quot;drift-$(date +%Y%m%d)&quot; \<br>            --log-events timestamp=$(date +%s000),message=&quot;$MESSAGE&quot; \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          echo &quot;✅ Drift event recorded in CloudWatch&quot;<br>          echo &quot;   • Metrics: 2 metrics sent&quot;<br>          echo &quot;   • Logs: Logged to /attendanceapp/drift-detection&quot;<br><br>      - name: Record No Drift in CloudWatch<br>        if: steps.drift.outputs.drift_detected == &#39;false&#39;<br>        run: |<br>          TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)<br>          <br>          echo &quot;📊 Recording clean state in CloudWatch...&quot;<br>          <br>          # Record that no drift was detected<br>          aws cloudwatch put-metric-data \<br>            --namespace &quot;AttendanceApp/Drift&quot; \<br>            --metric-name &quot;InfrastructureDrift&quot; \<br>            --value 0 \<br>            --dimensions Environment=dev,Type=terraform,Status=clean \<br>            --timestamp $TIMESTAMP \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          echo &quot;✅ Clean state recorded in CloudWatch&quot;<br><br>      - name: Create or Update GitHub Issue<br>        if: steps.drift.outputs.drift_detected == &#39;true&#39;<br>        env:<br>          GH_TOKEN: ${{ github.token }}<br>        run: |<br>          TODAY=$(date -u &#39;+%Y-%m-%d&#39;)<br>          <br>          # Create labels if missing<br>          gh label create infrastructure --color 0e8a16 --force 2&gt;/dev/null || true<br>          gh label create drift-detection --color d73a4a --force 2&gt;/dev/null || true<br>          gh label create needs-review --color fbca04 --force 2&gt;/dev/null || true<br>          <br>          # Check for existing issue<br>          EXISTING=$(gh issue list \<br>            --state open \<br>            --label drift-detection \<br>            --json number,title \<br>            --jq &quot;.[] | select(.title | contains(\&quot;$TODAY\&quot;)) | .number&quot; | head -1)<br>          <br>          if [ -n &quot;$EXISTING&quot; ]; then<br>            echo &quot;📝 Updating issue #$EXISTING&quot;<br>            cat &gt; comment.txt &lt;&lt; EOF<br>          ### 🔄 Drift Still Present<br>          <br>          **Check Time:** $(date -u)<br>          <br>          [View Run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})<br>          [View CloudWatch Logs](https://console.aws.amazon.com/cloudwatch/home?region=${{ env.AWS_REGION }}#logsV2:log-groups/log-group/\$252Fattendanceapp\$252Fdrift-detection)<br>          EOF<br>            gh issue comment $EXISTING --body-file comment.txt<br>            rm comment.txt<br>          else<br>            echo &quot;📝 Creating new issue&quot;<br>            cat &gt; body.txt &lt;&lt; EOF<br>          ## 🚨 Infrastructure Drift Detected<br>          <br>          **Detected:** $(date -u)<br>          **Environment:** dev<br>          **Region:** ${{ env.AWS_REGION }}<br>          <br>          ### Summary<br>          Infrastructure has diverged from Terraform state.<br>          <br>          ### Changes Detected<br>          See workflow run for full Terraform plan output.<br>          <br>          ### Next Steps<br>          1. Review changes in workflow run<br>          2. Check CloudWatch logs for drift history<br>          3. Update Terraform code or revert manual changes<br>          4. Run terraform apply to sync<br>          5. Close this issue<br>          <br>          ### Links<br>          - [📊 View Workflow Run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})<br>          - [📈 View CloudWatch Metrics](https://console.aws.amazon.com/cloudwatch/home?region=${{ env.AWS_REGION }}#metricsV2:graph=~();namespace=AttendanceApp/Drift)<br>          - [📋 View CloudWatch Logs](https://console.aws.amazon.com/cloudwatch/home?region=${{ env.AWS_REGION }}#logsV2:log-groups/log-group/\$252Fattendanceapp\$252Fdrift-detection)<br>          EOF<br>            <br>            gh issue create \<br>              --title &quot;🚨 Infrastructure Drift - $TODAY&quot; \<br>              --label infrastructure,drift-detection,needs-review \<br>              --body-file body.txt<br>            <br>            rm body.txt<br>          fi<br><br>      - name: Send Drift Email Notification<br>        if: steps.drift.outputs.drift_detected == &#39;true&#39;<br>        run: |<br>          RECIPIENTS=&quot;${{ secrets.EMAIL_RECIPIENTS }}&quot;<br>          RUN_URL=&quot;https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}&quot;<br>          CLOUDWATCH_LOGS=&quot;https://console.aws.amazon.com/cloudwatch/home?region=${{ env.AWS_REGION }}#logsV2:log-groups/log-group/\$252Fattendanceapp\$252Fdrift-detection&quot;<br>          <br>          # Create email HTML<br>          cat &gt; email.html &lt;&lt; &#39;HTMLEOF&#39;<br>          &lt;!DOCTYPE html&gt;<br>          &lt;html&gt;<br>          &lt;body style=&quot;font-family: Arial, sans-serif; margin: 0; padding: 0; background-color: #f4f4f4;&quot;&gt;<br>            &lt;div style=&quot;max-width: 600px; margin: 40px auto; background: white; border-radius: 8px; box-shadow: 0 2px 4px rgba(0,0,0,0.1); overflow: hidden;&quot;&gt;<br>              <br>              &lt;!-- Header --&gt;<br>              &lt;div style=&quot;background: #ff6b6b; color: white; padding: 40px 20px; text-align: center;&quot;&gt;<br>                &lt;h1 style=&quot;margin: 0; font-size: 28px;&quot;&gt;🚨 Infrastructure Drift Detected&lt;/h1&gt;<br>                &lt;p style=&quot;margin: 15px 0 0 0; font-size: 16px; opacity: 0.95;&quot;&gt;Dev Environment&lt;/p&gt;<br>              &lt;/div&gt;<br>              <br>              &lt;!-- Content --&gt;<br>              &lt;div style=&quot;padding: 40px 30px;&quot;&gt;<br>                &lt;p style=&quot;font-size: 16px; color: #333; line-height: 1.6; margin-top: 0;&quot;&gt;<br>                  Infrastructure has diverged from Terraform state.<br>                &lt;/p&gt;<br>                <br>                &lt;!-- Alert Box --&gt;<br>                &lt;div style=&quot;background: #fff3cd; padding: 20px; border-radius: 6px; border-left: 4px solid #ffc107; margin: 25px 0;&quot;&gt;<br>                  &lt;h3 style=&quot;margin: 0 0 10px 0; color: #856404; font-size: 16px;&quot;&gt;⚠️ Action Required&lt;/h3&gt;<br>                  &lt;p style=&quot;margin: 0; color: #856404; font-size: 14px;&quot;&gt;<br>                    Manual changes have been made to infrastructure. Review and reconcile with Terraform code.<br>                  &lt;/p&gt;<br>                &lt;/div&gt;<br>                <br>                &lt;!-- Details --&gt;<br>                &lt;div style=&quot;background: #f8f9fa; padding: 20px; border-radius: 6px; margin: 25px 0;&quot;&gt;<br>                  &lt;h3 style=&quot;margin: 0 0 15px 0; color: #333; font-size: 16px;&quot;&gt;📋 Detection Details&lt;/h3&gt;<br>                  &lt;table style=&quot;width: 100%; border-collapse: collapse;&quot;&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Time:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px;&quot;&gt;TIME_PLACEHOLDER&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Environment:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px;&quot;&gt;dev&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                    &lt;tr&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #666; font-size: 14px;&quot;&gt;&lt;strong&gt;Region:&lt;/strong&gt;&lt;/td&gt;<br>                      &lt;td style=&quot;padding: 8px 0; color: #333; font-size: 14px;&quot;&gt;eu-west-2&lt;/td&gt;<br>                    &lt;/tr&gt;<br>                  &lt;/table&gt;<br>                &lt;/div&gt;<br>                <br>                &lt;!-- Action Buttons --&gt;<br>                &lt;div style=&quot;text-align: center; margin: 35px 0 20px 0;&quot;&gt;<br>                  &lt;a href=&quot;URL_PLACEHOLDER&quot; <br>                     style=&quot;display: inline-block; background: #4CAF50; color: white; padding: 14px 28px; text-decoration: none; border-radius: 6px; margin: 5px; font-weight: 500; font-size: 15px;&quot;&gt;<br>                    📊 View Workflow<br>                  &lt;/a&gt;<br>                  &lt;a href=&quot;CLOUDWATCH_PLACEHOLDER&quot; <br>                     style=&quot;display: inline-block; background: #FF9800; color: white; padding: 14px 28px; text-decoration: none; border-radius: 6px; margin: 5px; font-weight: 500; font-size: 15px;&quot;&gt;<br>                    📈 View Logs<br>                  &lt;/a&gt;<br>                &lt;/div&gt;<br>              &lt;/div&gt;<br>              <br>              &lt;!-- Footer --&gt;<br>              &lt;div style=&quot;background: #f8f9fa; padding: 20px 30px; text-align: center; border-top: 1px solid #dee2e6;&quot;&gt;<br>                &lt;p style=&quot;margin: 0; font-size: 13px; color: #6c757d;&quot;&gt;<br>                  🤖 Automated drift detection from GitHub Actions<br>                &lt;/p&gt;<br>                &lt;p style=&quot;margin: 8px 0 0 0; font-size: 12px; color: #adb5bd;&quot;&gt;<br>                  Runs every hour to detect infrastructure changes<br>                &lt;/p&gt;<br>              &lt;/div&gt;<br>              <br>            &lt;/div&gt;<br>          &lt;/body&gt;<br>          &lt;/html&gt;<br>          HTMLEOF<br>          <br>          # Replace placeholders<br>          sed -i &quot;s|TIME_PLACEHOLDER|$(date -u &#39;+%Y-%m-%d %H:%M:%S UTC&#39;)|g&quot; email.html<br>          sed -i &quot;s|URL_PLACEHOLDER|$RUN_URL|g&quot; email.html<br>          sed -i &quot;s|CLOUDWATCH_PLACEHOLDER|$CLOUDWATCH_LOGS|g&quot; email.html<br>          <br>          # Convert HTML to JSON-safe string<br>          BODY_JSON=$(jq -Rs . &lt; email.html)<br>          <br>          # Send email via AWS SES<br>          aws ses send-email \<br>            --from &quot;tobi.akinlade.co@gmail.com&quot; \<br>            --destination &quot;ToAddresses=$RECIPIENTS&quot; \<br>            --message &quot;{<br>              \&quot;Subject\&quot;: {<br>                \&quot;Data\&quot;: \&quot;🚨 Infrastructure Drift Detected - $(date -u &#39;+%H:%M UTC&#39;)\&quot;,<br>                \&quot;Charset\&quot;: \&quot;UTF-8\&quot;<br>              },<br>              \&quot;Body\&quot;: {<br>                \&quot;Html\&quot;: {<br>                  \&quot;Data\&quot;: ${BODY_JSON},<br>                  \&quot;Charset\&quot;: \&quot;UTF-8\&quot;<br>                }<br>              }<br>            }&quot; \<br>            --region ${{ env.AWS_REGION }}<br>          <br>          # Clean up<br>          rm -f email.html<br>          echo &quot;✅ Drift notification sent to team&quot;<br><br>      - name: Summary<br>        run: |<br>          if [ &quot;${{ steps.drift.outputs.drift_detected }}&quot; == &quot;true&quot; ]; then<br>            echo &quot;### 🚨 Drift Detected!&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Time:** $(date -u)&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Exit Code:** ${{ steps.plan.outputs.exit_code }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Actions Taken:**&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;- ✅ CloudWatch metrics recorded&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;- ✅ CloudWatch logs created&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;- ✅ GitHub issue created/updated&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;- ✅ Email notification sent&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Next Check:** $(date -u -d &#39;+1 hour&#39; &#39;+%H:00 UTC&#39;)&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          else<br>            echo &quot;### ✅ No Drift Detected&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Time:** $(date -u)&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Exit Code:** ${{ steps.plan.outputs.exit_code }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Status:** Infrastructure matches Terraform state&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;- ✅ Clean state recorded in CloudWatch&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>            echo &quot;**Next Check:** $(date -u -d &#39;+1 hour&#39; &#39;+%H:00 UTC&#39;)&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          fi</pre><p>Commit the code</p><pre>git add .<br>git commit -m &#39;terraform drift detection&#39;<br>git push origin main</pre><h3>Testing Drift Detection</h3><pre># Create drift manually by changing ECS desired count<br>aws ecs update-service \<br>  --cluster attendance-app-dev-cluster \<br>  --service attendance-app-dev-service \<br>  --desired-count 5 \<br>  --region eu-west-2<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DR_z9KlUbTsh-ZjY4njozA.png" /></figure><pre>run the terraform-drift-detection workflow manually</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BK37edtMkG3Esm4IzE4blA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mDofnZ2N1S8imMkncbqqTA.png" /></figure><h4>Also captured in cloudwatch log</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLNfQ6yUDzbjeU4Uwrix0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RmxTHTIsnJFnjPELrCc_2A.png" /></figure><h3>Resolving Drift</h3><p><strong>Option A: Keep Manual Changes (Update Terraform)</strong></p><pre># Update Terraform to match manual changes<br>aws ecs update-service \<br>  --cluster attendance-app-dev-cluster \<br>  --service attendance-app-dev-service \<br>  --desired-count 2 \<br>  --region eu-west-2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KZQE7VL3Wcr3YMfQA0W3kw.png" /></figure><h3>Component 4: Health Monitoring</h3><h3>What It Is</h3><p>Automated health checks that run <strong>every 15 minutes</strong> to verify your services are running properly, completely independent of CloudWatch Alarms.</p><h3>Why You Need It (When Alarms Aren’t Enough)</h3><p><strong>The Gap Alarms Miss:</strong></p><p>CloudWatch Alarms are reactive — they wait for metrics to breach thresholds. But some problems don’t show up in metrics:</p><pre>Scenario 1: Application is running but not responding<br>- CPU: ✅ Normal<br>- Memory: ✅ Normal  <br>- Tasks: ✅ Running<br>- Problem: App frozen, returning HTTP 500<br>Scenario 2: All targets marked healthy but app broken<br>- Health check endpoint: ✅ Returns 200<br>- Actual app functionality: ❌ Database disconnected<br>Scenario 3: Metrics delayed<br>- Problem occurs at 3:00 PM<br>- Metrics arrive at 3:05 PM<br>- Alarm triggers at 3:10 PM<br>- Health check would have caught it at 3:00 PM</pre><h3><strong>Why Every 15 Minutes?</strong></h3><p>Every minute? Too noisy and expensive.<br> ❌ Burns GitHub Actions minutes<br> ❌ Causes false alarms<br> ❌ Risks AWS API rate limits</p><p>Every hour? Too slow.<br> ❌ Issues linger<br> ❌ Users suffer longer</p><p>Every 15 minutes hits the balance:<br> ✅ Catches problems fast (max 15-min impact)<br> ✅ Stays cheap (96 min/day ≈ free tier)<br> ✅ Avoids alert fatigue<br> ✅ Balances speed, cost, and sanity</p><h3>Implementation</h3><p>Create .github/workflows/observability.yml:</p><pre>name: Infrastructure Health Check<br><br>on:<br>  schedule:<br>    - cron: &#39;*/15 * * * *&#39;  # Every 15 minutes<br>  workflow_dispatch:<br><br>env:<br>  AWS_REGION: eu-west-2<br>  CLUSTER_NAME: attendance-app-dev-cluster<br>  SERVICE_NAME: attendance-app-dev-service<br><br>jobs:<br>  health-check:<br>    name: Check Infrastructure Health<br>    runs-on: ubuntu-latest<br>    permissions:<br>      id-token: write<br>      contents: read<br>    <br>    steps:<br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br>          aws-region: ${{ env.AWS_REGION }}<br><br>      - name: Check ECS Service Health<br>        id: ecs<br>        run: |<br>          SERVICE_INFO=$(aws ecs describe-services \<br>            --cluster $CLUSTER_NAME \<br>            --services $SERVICE_NAME \<br>            --query &#39;services[0]&#39;)<br>          <br>          RUNNING=$(echo $SERVICE_INFO | jq -r &#39;.runningCount&#39;)<br>          DESIRED=$(echo $SERVICE_INFO | jq -r &#39;.desiredCount&#39;)<br>          PENDING=$(echo $SERVICE_INFO | jq -r &#39;.pendingCount&#39;)<br>          <br>          echo &quot;running=$RUNNING&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;desired=$DESIRED&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;pending=$PENDING&quot; &gt;&gt; $GITHUB_OUTPUT<br>          <br>          if [ &quot;$RUNNING&quot; -eq &quot;$DESIRED&quot; ] &amp;&amp; [ &quot;$PENDING&quot; -eq &quot;0&quot; ]; then<br>            echo &quot;status=healthy&quot; &gt;&gt; $GITHUB_OUTPUT<br>          else<br>            echo &quot;status=degraded&quot; &gt;&gt; $GITHUB_OUTPUT<br>          fi<br><br>      - name: Check ALB Target Health<br>        id: alb<br>        run: |<br>          TG_ARN=$(aws elbv2 describe-target-groups \<br>            --names attendance-app-dev-tg \<br>            --query &#39;TargetGroups[0].TargetGroupArn&#39; \<br>            --output text)<br>          <br>          HEALTH=$(aws elbv2 describe-target-health \<br>            --target-group-arn $TG_ARN \<br>            --query &#39;TargetHealthDescriptions[*].TargetHealth.State&#39; \<br>            --output json)<br>          <br>          HEALTHY=$(echo $HEALTH | jq -r &#39;. | map(select(. == &quot;healthy&quot;)) | length&#39;)<br>          TOTAL=$(echo $HEALTH | jq -r &#39;. | length&#39;)<br>          <br>          echo &quot;healthy=$HEALTHY&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;total=$TOTAL&quot; &gt;&gt; $GITHUB_OUTPUT<br>          <br>          if [ &quot;$HEALTHY&quot; -eq &quot;$TOTAL&quot; ] &amp;&amp; [ &quot;$TOTAL&quot; -gt &quot;0&quot; ]; then<br>            echo &quot;status=healthy&quot; &gt;&gt; $GITHUB_OUTPUT<br>          else<br>            echo &quot;status=unhealthy&quot; &gt;&gt; $GITHUB_OUTPUT<br>          fi<br><br>      - name: Check RDS Health<br>        id: rds<br>        run: |<br>          RDS_STATUS=$(aws rds describe-db-instances \<br>            --db-instance-identifier attendance-app-dev-db \<br>            --query &#39;DBInstances[0].DBInstanceStatus&#39; \<br>            --output text)<br>          <br>          echo &quot;status=$RDS_STATUS&quot; &gt;&gt; $GITHUB_OUTPUT<br>          <br>          if [ &quot;$RDS_STATUS&quot; == &quot;available&quot; ]; then<br>            echo &quot;healthy=true&quot; &gt;&gt; $GITHUB_OUTPUT<br>          else<br>            echo &quot;healthy=false&quot; &gt;&gt; $GITHUB_OUTPUT<br>          fi<br><br>      - name: Get CloudWatch Metrics<br>        id: metrics<br>        run: |<br>          # Get last 5 minutes of CPU usage<br>          END_TIME=$(date -u +%Y-%m-%dT%H:%M:%S)<br>          START_TIME=$(date -u -d &#39;5 minutes ago&#39; +%Y-%m-%dT%H:%M:%S)<br>          <br>          CPU=$(aws cloudwatch get-metric-statistics \<br>            --namespace AWS/ECS \<br>            --metric-name CPUUtilization \<br>            --dimensions Name=ServiceName,Value=$SERVICE_NAME Name=ClusterName,Value=$CLUSTER_NAME \<br>            --start-time $START_TIME \<br>            --end-time $END_TIME \<br>            --period 300 \<br>            --statistics Average \<br>            --query &#39;Datapoints[0].Average&#39; \<br>            --output text)<br>          <br>          MEMORY=$(aws cloudwatch get-metric-statistics \<br>            --namespace AWS/ECS \<br>            --metric-name MemoryUtilization \<br>            --dimensions Name=ServiceName,Value=$SERVICE_NAME Name=ClusterName,Value=$CLUSTER_NAME \<br>            --start-time $START_TIME \<br>            --end-time $END_TIME \<br>            --period 300 \<br>            --statistics Average \<br>            --query &#39;Datapoints[0].Average&#39; \<br>            --output text)<br>          <br>          echo &quot;cpu=${CPU:-0}&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;memory=${MEMORY:-0}&quot; &gt;&gt; $GITHUB_OUTPUT<br><br>      - name: Create Health Report<br>        run: |<br>          cat &gt; health_report.html &lt;&lt; EOF<br>          &lt;!DOCTYPE html&gt;<br>          &lt;html&gt;<br>          &lt;head&gt;<br>            &lt;style&gt;<br>              body { font-family: Arial, sans-serif; max-width: 800px; margin: 0 auto; }<br>              .header { background: #4CAF50; color: white; padding: 20px; text-align: center; }<br>              .status { display: flex; justify-content: space-around; margin: 20px 0; }<br>              .status-card { background: #f9f9f9; padding: 20px; border-radius: 8px; flex: 1; margin: 0 10px; text-align: center; }<br>              .healthy { border-left: 4px solid #4CAF50; }<br>              .unhealthy { border-left: 4px solid #f44336; }<br>              .metric { display: flex; justify-content: space-between; padding: 10px 0; border-bottom: 1px solid #ddd; }<br>            &lt;/style&gt;<br>          &lt;/head&gt;<br>          &lt;body&gt;<br>            &lt;div class=&quot;header&quot;&gt;<br>              &lt;h1&gt;🔍 Infrastructure Health Report&lt;/h1&gt;<br>              &lt;p&gt;$(date -u &#39;+%Y-%m-%d %H:%M:%S UTC&#39;)&lt;/p&gt;<br>            &lt;/div&gt;<br>            <br>            &lt;div class=&quot;status&quot;&gt;<br>              &lt;div class=&quot;status-card ${{ steps.ecs.outputs.status == &#39;healthy&#39; &amp;&amp; &#39;healthy&#39; || &#39;unhealthy&#39; }}&quot;&gt;<br>                &lt;h3&gt;ECS Service&lt;/h3&gt;<br>                &lt;p&gt;&lt;strong&gt;Status:&lt;/strong&gt; ${{ steps.ecs.outputs.status }}&lt;/p&gt;<br>                &lt;p&gt;Running: ${{ steps.ecs.outputs.running }}/${{ steps.ecs.outputs.desired }}&lt;/p&gt;<br>              &lt;/div&gt;<br>              <br>              &lt;div class=&quot;status-card ${{ steps.alb.outputs.status == &#39;healthy&#39; &amp;&amp; &#39;healthy&#39; || &#39;unhealthy&#39; }}&quot;&gt;<br>                &lt;h3&gt;Load Balancer&lt;/h3&gt;<br>                &lt;p&gt;&lt;strong&gt;Status:&lt;/strong&gt; ${{ steps.alb.outputs.status }}&lt;/p&gt;<br>                &lt;p&gt;Healthy Targets: ${{ steps.alb.outputs.healthy }}/${{ steps.alb.outputs.total }}&lt;/p&gt;<br>              &lt;/div&gt;<br>              <br>              &lt;div class=&quot;status-card ${{ steps.rds.outputs.healthy == &#39;true&#39; &amp;&amp; &#39;healthy&#39; || &#39;unhealthy&#39; }}&quot;&gt;<br>                &lt;h3&gt;RDS Database&lt;/h3&gt;<br>                &lt;p&gt;&lt;strong&gt;Status:&lt;/strong&gt; ${{ steps.rds.outputs.status }}&lt;/p&gt;<br>              &lt;/div&gt;<br>            &lt;/div&gt;<br>            <br>            &lt;h2&gt;Metrics&lt;/h2&gt;<br>            &lt;div class=&quot;metric&quot;&gt;<br>              &lt;strong&gt;CPU Utilization:&lt;/strong&gt;<br>              &lt;span&gt;${{ steps.metrics.outputs.cpu }}%&lt;/span&gt;<br>            &lt;/div&gt;<br>            &lt;div class=&quot;metric&quot;&gt;<br>              &lt;strong&gt;Memory Utilization:&lt;/strong&gt;<br>              &lt;span&gt;${{ steps.metrics.outputs.memory }}%&lt;/span&gt;<br>            &lt;/div&gt;<br>          &lt;/body&gt;<br>          &lt;/html&gt;<br>          EOF<br>          <br>          cat health_report.html<br><br>      - name: Send Alert if Unhealthy<br>        if: steps.ecs.outputs.status != &#39;healthy&#39; || steps.alb.outputs.status != &#39;healthy&#39; || steps.rds.outputs.healthy != &#39;true&#39;<br>        run: |<br>          BODY=$(cat health_report.html | tr -d &#39;\n&#39;)<br>          <br>          aws ses send-email \<br>            --from &quot;Infrastructure Monitor &lt;tobi.akinlade.co@gmail.com&gt;&quot; \<br>            --destination &quot;ToAddresses=${{ secrets.EMAIL_RECIPIENTS }}&quot; \<br>            --message &quot;Subject={Data=&#39;⚠️ Infrastructure Health Alert&#39;,Charset=utf8},Body={Html={Data=&#39;$BODY&#39;,Charset=utf8}}&quot; \<br>            --region ${{ env.AWS_REGION }}<br><br>      - name: Summary<br>        run: |<br>          echo &quot;### 🔍 Health Check Summary&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Time:** $(date -u)&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;| Component | Status | Details |&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;|-----------|--------|---------|&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;| ECS Service | ${{ steps.ecs.outputs.status }} | ${{ steps.ecs.outputs.running }}/${{ steps.ecs.outputs.desired }} tasks |&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;| Load Balancer | ${{ steps.alb.outputs.status }} | ${{ steps.alb.outputs.healthy }}/${{ steps.alb.outputs.total }} healthy |&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;| RDS Database | ${{ steps.rds.outputs.status }} | - |&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Metrics:**&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;- CPU: ${{ steps.metrics.outputs.cpu }}%&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;- Memory: ${{ steps.metrics.outputs.memory }}%&quot; &gt;&gt; $GITHUB_STEP_SUMMARY</pre><pre>commit the code to github</pre><pre>git add .<br>git commit -m &#39;observability check&#39;<br>git push origin main</pre><h3>Testing Health Monitoring</h3><pre># Trigger health check manually<br>run the workflow manually</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tv-Yrf3MNMrmu5DY0dILA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZWaXw5qTwUOjpul2-wvmg.png" /></figure><p>If there is a problem, an email will be sent to the team’s email or the engineers on the on-call rota that looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fD62g6G--s7lHsw5M1DLVA.png" /></figure><h3>What’s Next</h3><p>Congratulations! You’ve built a production-grade observability stack. Here’s what to explore next:</p><h3>Clean up</h3><p>On both observability-check and the terraform-drift-detection workflow, unschedule the cron job</p><pre>#comment out this part and commit your work back to git<br>  schedule:<br>    - cron: &#39;*/15 * * * *&#39;  # Every 15 minutes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*crWJvmYeJbhKFmMdtBCY-w.png" /></figure><p>Run the Terraform destroy workflow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C1Mjd53DNQpyahb0wKxe7A.png" /></figure><h3>The Core Philosophy</h3><p><strong>You can’t manage what you can’t measure.</strong></p><ul><li><strong>Visibility</strong> comes first</li><li><strong>Automation</strong> makes it sustainable</li><li><strong>Proactivity</strong> prevents firefighting</li><li><strong>Documentation</strong> enables learning</li></ul><h3>Production Readiness Checklist</h3><p>Before calling your infrastructure “production-ready”:</p><ul><li>✅ Monitoring all critical components</li><li>✅ Alarms on failure conditions</li><li>✅ Notifications to team</li><li>✅ Drift detection for changes</li><li>✅ Health checks every 15 minutes</li><li>✅ Historical data for debugging</li><li>✅ Runbooks documented</li><li>✅ Team trained on tools</li></ul><h3>🧠 Final Thoughts</h3><p>Observability isn’t about fancy dashboards or expensive tools — it’s about <strong>understanding your system</strong>.</p><p>💡 It’s knowing what matters, measuring it, alerting smartly, and responding fast when things break.<br> AWS gives you the building blocks — now you’ve connected them into something powerful.</p><h3>🎯 You Started with Nothing. Now You Have:</h3><ul><li>✅ <strong>Production infrastructure</strong> (Part 1)</li><li>✅ <strong>Automated deployments</strong> (Part 2)</li><li>✅ <strong>Complete observability</strong> (Part 3)</li></ul><p><strong>This is production-ready.</strong> 🎉</p><h3>🚀 What You’ve Achieved</h3><p>Your system now:</p><ul><li>Monitors itself 24/7</li><li>Alerts you <em>before</em> customers notice issues</li><li>Detects unauthorised changes automatically</li><li>Recovers gracefully from failures</li></ul><p>You’ve built the foundation of a reliable, modern production environment.</p><h3>🧩 This Series</h3><ul><li><a href="https://medium.com/@tobi.akinlade.co/automating-aws-ecs-deployment-part-1-infrastructure-as-code-with-terraform-9ce9e69d6436">Part 1 — Infrastructure Foundation</a></li><li><a href="https://medium.com/@tobi.akinlade.co/automating-aws-ecs-deployment-part-2-ci-cd-pipeline-with-github-actions-66082cc70f17">Part 2 — CI/CD Automation</a></li><li><a href="https://medium.com/@tobi.akinlade.co/a7e32aa3ff01">Part 3 — Production Observability</a> ← <em>You’re here</em></li></ul><h3>⛵ What’s Next: Kubernetes Projects</h3><p>We’ve conquered AWS ECS — next, we’ll go deeper into <strong>Kubernetes projects</strong>:<br> real-world deployments, autoscaling, observability, GitOps, and more.</p><p>Follow me here or on <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">LinkedIn</a> to get notified when the new series drops.</p><p>Now go <strong>deploy with confidence.</strong> 🚀</p><blockquote><em>🕒 </em>The best time to build observability was before your first deployment.<br> The second best time is right now.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7e32aa3ff01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating AWS ECS Deployment — Part 2: CI/CD Pipeline with GitHub Actions]]></title>
            <link>https://medium.com/@tobi.akinlade.co/automating-aws-ecs-deployment-part-2-ci-cd-pipeline-with-github-actions-66082cc70f17?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/66082cc70f17</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Thu, 16 Oct 2025 20:50:08 GMT</pubDate>
            <atom:updated>2025-10-16T20:50:08.987Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Automating AWS ECS Deployment — Part 2: CI/CD Pipeline with GitHub Actions</strong></h3><h3>From Manual Deployments to Fully Automated GitOps</h3><p>In <a href="https://medium.com/@tobi.akinlade.co/automating-aws-ecs-deployment-part-1-infrastructure-as-code-with-terraform-9ce9e69d6436">Part 1: Infrastructure as Code with Terraform</a>, we built a complete 3-tier infrastructure on AWS using Terraform. Now comes the exciting part: automating the entire application deployment process so that every code commit automatically flows through testing, security scanning, and deployment — all without manual intervention.</p><p>By the end of this guide, you’ll have a production-ready CI/CD pipeline where pushing code to GitHub triggers an automated workflow that builds, tests, scans, and deploys your application to AWS ECS in minutes.</p><h3>What We’ll Build</h3><p>A complete CI/CD pipeline that:</p><ul><li>✅ Automatically triggers on every code push to main branch</li><li>✅ Runs automated tests and code quality checks</li><li>✅ Performs security vulnerability scanning</li><li>✅ Builds and tags Docker images with commit SHA</li><li>✅ Pushes images to Amazon ECR</li><li>✅ Deploys new versions to ECS with zero downtime</li><li>✅ Uses AWS OIDC for secure authentication (no access keys!)</li><li>✅ Provides deployment summaries and status notifications</li></ul><h3>Prerequisites</h3><p>Before starting, ensure you have:</p><ul><li>✅ Completed Part 1 (infrastructure deployed via Terraform)</li><li>✅ A GitHub repository for your application</li><li>✅ Basic understanding of Git and GitHub</li><li>✅ The Flask attendance application running on ECS</li></ul><h3>Architecture Overview</h3><p>Here’s how the automated CI/CD pipeline works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LrA0J_7PuSwdLg8CQX0Pdw.png" /></figure><h3><strong>Clone the Repo</strong></h3><pre>git clone https://github.com/tobiakinlade/attendance-app.git</pre><h3>Part A: AWS OIDC Authentication Setup</h3><h3>Step 1: Understanding OIDC vs Access Keys</h3><p><strong>What we’re doing:</strong> Setting up OpenID Connect (OIDC) authentication so GitHub Actions can securely access AWS without storing long-lived credentials.</p><p><strong>Why this matters:</strong></p><ul><li><strong>No Secret Storage</strong>: No access keys stored in GitHub (much more secure)</li><li><strong>Temporary Credentials</strong>: GitHub gets short-lived tokens that expire</li><li><strong>Better Security</strong>: If GitHub is compromised, attackers can’t use credentials</li><li><strong>Audit Trail</strong>: Every action is logged with clear identity</li><li><strong>Industry Best Practice</strong>: This is how modern cloud deployments should work</li></ul><p><strong>The old way (dangerous):</strong></p><pre># DON&#39;T DO THIS!<br>aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}</pre><p><strong>The new way (secure):</strong></p><pre># DO THIS!<br>role-to-assume: ${{ secrets.AWS_ROLE_ARN }}</pre><h3>Step 2: Create OIDC Identity Provider</h3><p><strong>What we’re doing:</strong> Telling AWS to trust GitHub’s identity system.</p><p><strong>Why this matters:</strong> This establishes the trust relationship between GitHub and AWS, allowing GitHub to request temporary credentials.</p><pre>aws iam create-open-id-connect-provider \<br>  --url https://token.actions.githubusercontent.com \<br>  --client-id-list sts.amazonaws.com \<br>  --thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1 \<br>  --region eu-west-2</pre><p><strong>Verify it was created:</strong></p><pre>aws iam list-open-id-connect-providers</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3In5Q5zoqLZ5Fpk4HvleA.png" /></figure><h3>Step 3: Create IAM Role with Trust Policy</h3><p><strong>What we’re doing:</strong> Creating an IAM role that GitHub Actions can assume, but ONLY for your specific repository.</p><p><strong>Why this matters:</strong> This ensures that only YOUR repository can deploy to YOUR AWS account. Even if someone forks your repo, they can’t deploy to your infrastructure.</p><p>Create github-oidc-trust-policy.json:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Federated&quot;: &quot;arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com&quot;<br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;token.actions.githubusercontent.com:aud&quot;: &quot;sts.amazonaws.com&quot;<br>        },<br>        &quot;StringLike&quot;: {<br>          &quot;token.actions.githubusercontent.com:sub&quot;: &quot;repo:YOUR_GITHUB_USERNAME/YOUR_REPO_NAME:*&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p><strong>Replace:</strong></p><ul><li>YOUR_ACCOUNT_ID with your AWS account ID</li><li>YOUR_GITHUB_USERNAME/YOUR_REPO_NAME with your repository</li></ul><p>Create the role:</p><pre>aws iam create-role \<br>  --role-name GitHubActionsECSDeployRole \<br>  --assume-role-policy-document file://github-oidc-trust-policy.json \<br>  --description &quot;Role for GitHub Actions to deploy to ECS&quot;</pre><h3>Step 4: Attach Required Permissions</h3><p><strong>What we’re doing:</strong> Giving the role permission to push Docker images to ECR and deploy to ECS.</p><p><strong>Why this matters:</strong> The role needs specific permissions to:</p><ul><li>Authenticate with ECR</li><li>Push container images</li><li>Update ECS services</li><li>Register new task definitions</li></ul><p>Create github-actions-permissions.json:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;ecr:GetAuthorizationToken&quot;,<br>        &quot;ecr:BatchCheckLayerAvailability&quot;,<br>        &quot;ecr:GetDownloadUrlForLayer&quot;,<br>        &quot;ecr:BatchGetImage&quot;,<br>        &quot;ecr:PutImage&quot;,<br>        &quot;ecr:InitiateLayerUpload&quot;,<br>        &quot;ecr:UploadLayerPart&quot;,<br>        &quot;ecr:CompleteLayerUpload&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;*&quot;<br>    },<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;ecs:UpdateService&quot;,<br>        &quot;ecs:DescribeServices&quot;,<br>        &quot;ecs:DescribeTaskDefinition&quot;,<br>        &quot;ecs:RegisterTaskDefinition&quot;,<br>        &quot;ecs:ListTaskDefinitions&quot;,<br>        &quot;ecs:DescribeTasks&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;*&quot;<br>    },<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [&quot;iam:PassRole&quot;],<br>      &quot;Resource&quot;: &quot;arn:aws:iam::*:role/*ecs*&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringLike&quot;: {<br>          &quot;iam:PassedToService&quot;: &quot;ecs-tasks.amazonaws.com&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>Create and attach the policy:</p><pre># Create the policy<br>aws iam create-policy \<br>  --policy-name GitHubActionsECSDeployPolicy \<br>  --policy-document file://github-actions-permissions.json<br><br># Attach it to the role<br>aws iam attach-role-policy \<br>  --role-name GitHubActionsECSDeployRole \<br>  --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/GitHubActionsECSDeployPolicy</pre><h3>Step 5: Configure GitHub Secrets</h3><p><strong>What we’re doing:</strong> Storing the IAM role ARN as a GitHub secret so our workflow can use it.</p><p><strong>Why this matters:</strong> Secrets are encrypted in GitHub and only accessible to workflow runs. This keeps your AWS role ARN secure.</p><p><strong>Get your role ARN:</strong></p><pre># Get the role ARN (you&#39;ll need this for GitHub secrets)<br>aws iam get-role \<br>  --role-name GitHubActionsECSRole \<br>  --query &#39;Role.Arn&#39; \<br>  --output text<br>```<br><br>**Copy this ARN!** It should look like:<br>```<br>arn:aws:iam::180048382895:role/GitHubActionsECSRole<br>```<br><br>### Add GitHub Secrets<br><br>Go to your GitHub repository:<br>1. **Settings** → **Secrets and variables** → **Actions**<br>2. Click **&quot;New repository secret&quot;**<br>3. Add these secrets:<br>```<br>Name: AWS_ROLE_ARN<br>Value: arn:aws:iam::180048382895:role/GitHubActionsECSRole<br><br>Name: AWS_REGION<br>Value: eu-west-2<br><br>Name: ECR_REPOSITORY<br>Value: attendance-app-dev<br><br>Name: ECS_CLUSTER<br>Value: attendance-app-dev-cluster<br><br>Name: ECS_SERVICE<br>Value: attendance-app-dev-service<br><br>Name: CONTAINER_NAME<br>Value: attendance-app-container<br><br>Name: DB_USERNAME<br>Value: dbadmin<br><br>Name: DB_PASSWORD<br>Value: ChangeMe123!SecurePassword</pre><p><strong>Verify secrets:</strong></p><pre>gh secret list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fqLAo-_LrGt13EBipmT_-A.png" /></figure><h3>Part B: The CI/CD Pipeline</h3><h3>Step 6: Understanding the Pipeline Structure</h3><p><strong>What we’re doing:</strong> Creating a multi-stage pipeline that tests, builds, and deploys our application automatically.</p><p><strong>Why this matters:</strong></p><ul><li><strong>Quality Assurance</strong>: Catches bugs before they reach production</li><li><strong>Security</strong>: Identifies vulnerabilities early</li><li><strong>Consistency</strong>: Every deployment follows the same proven process</li><li><strong>Speed</strong>: Automated deployments are faster than manual ones</li><li><strong>Confidence</strong>: Tests pass = safe to deploy</li></ul><p><strong>The Three Stages:</strong></p><ol><li><strong>Test Stage</strong> (Runs first, fails fast)</li></ol><ul><li>Lints code for style issues</li><li>Runs unit tests</li><li>Checks code coverage</li></ul><p><strong>2. Security Scan Stage</strong> (Runs in parallel with tests)</p><ul><li>Scans for known vulnerabilities</li><li>Checks dependencies for security issues</li><li>Identifies outdated packages</li></ul><p><strong>3. Build &amp; Deploy Stage</strong> (Only runs if tests pass)</p><ul><li>Builds Docker image</li><li>Tags with commit SHA</li><li>Pushes to ECR</li><li>Updates ECS service</li><li>Waits for healthy deployment</li></ul><h3>Step 7: Create the GitHub Actions Workflow</h3><p><strong>What we’re doing:</strong> Creating a workflow file that defines our entire CI/CD process.</p><p><strong>Why this matters:</strong> This single file automates everything from code commit to production deployment. Once set up, developers just push code and the pipeline handles the rest.</p><p>Create .github/workflows/deploy.yml:</p><pre>name: Build and Deploy to ECS<br><br>on:<br>  push:<br>    branches:<br>      - main<br>    paths:<br>      - &#39;app/**&#39;<br>      - &#39;Dockerfile&#39;<br>      - &#39;requirements.txt&#39;<br>      - &#39;.github/workflows/deploy.yml&#39;<br><br>env:<br>  AWS_REGION: eu-west-2<br>  ECR_REPOSITORY: attendance-app-dev  <br>  ECS_CLUSTER: attendance-app-dev-cluster<br>  ECS_SERVICE: attendance-app-dev-service<br>  CONTAINER_NAME: attendance-app-container<br><br>jobs:<br>  test:<br>    name: Run Tests<br>    runs-on: ubuntu-latest<br><br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Set up Python<br>        uses: actions/setup-python@v5<br>        with:<br>          python-version: &#39;3.11&#39;<br>          cache: &#39;pip&#39;<br>      <br>      - name: Install dependencies<br>        run: |<br>          python -m pip install --upgrade pip<br>          pip install -r requirements.txt<br><br>      - name: Install dev dependencies<br>        run: |<br>          pip install flake8 pytest pytest-cov<br>  <br>      - name: Lint with flake8<br>        continue-on-error: true  <br>        run: |<br>          # Stop build if there are Python syntax errors<br>          flake8 app --count --select=E9,F63,F7,F82 --show-source --statistics<br>          # Exit-zero treats all errors as warnings<br>          flake8 app --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics<br><br>      - name: Run tests<br>        run: |<br>          # If tests exist<br>          # pytest tests/ --cov=app --cov-report=xml<br>          echo &quot;✅ No tests defined yet - skipping&quot;<br>      <br>  security-scan:<br>    name: Security Scan<br>    runs-on: ubuntu-latest<br><br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Run Trivy vulnerability scanner<br>        uses: aquasecurity/trivy-action@master<br>        with:<br>          scan-type: &#39;fs&#39;<br>          scan-ref: &#39;.&#39;<br>          format: &#39;table&#39;<br>          exit-code: &#39;0&#39;<br>  <br>  build-and-deploy:<br>    name: Build and Deploy<br>    runs-on: ubuntu-latest<br>    needs: [test, security-scan]<br>    permissions:<br>      id-token: write<br>      contents: read<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br>      <br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br>          aws-region: ${{ env.AWS_REGION }}<br>    <br>      - name: Login to Amazon ECR<br>        id: login-ecr<br>        uses: aws-actions/amazon-ecr-login@v2<br>      <br>      - name: Build, tag, push image to Amazon ECR<br>        id: build-image<br>        env:<br>          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}<br>          IMAGE_TAG: ${{ github.sha }}<br>        run: |<br>          # Build Docker image<br>          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .<br>          docker tag $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG $ECR_REGISTRY/$ECR_REPOSITORY:latest  <br><br>          # Push both tags<br>          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG<br>          docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest<br><br>          # Output image URI for deployment<br>          echo &quot;image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG&quot; &gt;&gt; $GITHUB_OUTPUT<br><br>      - name: Download task definition<br>        run: |<br>          aws ecs describe-task-definition \<br>            --task-definition attendance-app-dev \<br>            --query taskDefinition &gt; task-definition.json<br>      <br>      - name: Update ECS task definition<br>        id: task-def<br>        uses: aws-actions/amazon-ecs-render-task-definition@v1<br>        with: <br>          task-definition: task-definition.json<br>          container-name: ${{ env.CONTAINER_NAME }}<br>          image: ${{ steps.build-image.outputs.image }}<br><br>      - name: Deploy to Amazon ECS<br>        uses: aws-actions/amazon-ecs-deploy-task-definition@v1<br>        with:<br>          task-definition: ${{ steps.task-def.outputs.task-definition }}<br>          service: ${{ env.ECS_SERVICE }}<br>          cluster: ${{ env.ECS_CLUSTER }}<br>          wait-for-service-stability: true<br><br>      - name: Deployment Summary<br>        run: |<br>          echo &quot;### 🚀 Deployment Successful!&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Image:** ${{ steps.build-image.outputs.image }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Cluster:** ${{ env.ECS_CLUSTER }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Service:** ${{ env.ECS_SERVICE }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Commit:** ${{ github.sha }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY</pre><h3>Step 8: Understanding Each Job in Detail</h3><h4>Job 1: Test Stage</h4><p><strong>What we’re doing:</strong> Running automated tests and code quality checks before allowing any deployment.</p><p><strong>Why this matters:</strong></p><ul><li><strong>Catches Bugs Early</strong>: Find issues before they reach production</li><li><strong>Code Quality</strong>: Ensures code meets style standards</li><li><strong>Confidence</strong>: Green tests = safe to deploy</li><li><strong>Fast Feedback</strong>: Developers know within minutes if their code works</li></ul><p><strong>Key components:</strong></p><ol><li><strong>Python Cache</strong> (cache: &#39;pip&#39;)</li></ol><ul><li>Caches pip packages between runs</li><li>Makes subsequent builds 3–5x faster</li><li>Saves time and GitHub Actions minutes</li></ul><p><strong>2. Flake8 Linting</strong></p><ul><li>Checks Python code style (PEP 8)</li><li>Finds common programming errors</li><li>Enforces consistency across the codebase</li></ul><p><strong>3. Pytest with Coverage</strong></p><ul><li>Runs your test suite</li><li>Measures what percentage of code is tested</li><li>Generates coverage reports for visibility</li></ul><h4>Job 2: Security Scan</h4><p><strong>What we’re doing:</strong> Scanning code and dependencies for known security vulnerabilities.</p><p><strong>Why this matters:</strong></p><ul><li><strong>Proactive Security</strong>: Find vulnerabilities before attackers do</li><li><strong>Compliance</strong>: Many regulations require vulnerability scanning</li><li><strong>Dependency Monitoring</strong>: Libraries have vulnerabilities too</li><li><strong>Risk Reduction</strong>: Know what you’re deploying</li></ul><p><strong>Trivy scanner finds:</strong></p><ul><li>Known CVEs in dependencies</li><li>Outdated packages with security fixes</li><li>Misconfigurations in Dockerfiles</li><li>Hardcoded secrets (passwords, API keys)</li></ul><h4>Job 3: Build and Deploy</h4><p><strong>What we’re doing:</strong> Building a Docker image, pushing it to ECR, and deploying to ECS.</p><p><strong>Why this matters:</strong></p><ul><li><strong>Immutable Deployments</strong>: Every deployment is a specific, tagged image</li><li><strong>Rollback Capability</strong>: Can always go back to previous image</li><li><strong>Traceability</strong>: Each image tagged with commit SHA</li><li><strong>Zero Downtime</strong>: ECS rolls out new tasks gradually</li></ul><p><strong>Key steps explained:</strong></p><ol><li><strong>Configure AWS credentials (OIDC)</strong></li></ol><pre>- name: Configure AWS credentials<br>     uses: aws-actions/configure-aws-credentials@v4<br>     with:<br>       role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br></pre><ul><li>Exchanges GitHub’s identity token for AWS credentials</li><li>Credentials expire after 1 hour (secure!)</li><li>No long-lived secrets stored anywhere</li></ul><p><strong>2. Build and Tag</strong></p><pre>IMAGE_TAG: ${{ github.sha }}</pre><ul><li>Tags image with commit SHA (e.g., abc123def456)</li><li>Also tags as latest for convenience</li><li>Makes it easy to identify exactly what’s deployed</li></ul><p><strong>3. Update Task Definition</strong></p><ul><li>Downloads current ECS task definition</li><li>Updates only the image URI</li><li>Registers new revision</li><li>Keeps all other settings unchanged</li></ul><p><strong>4. Deploy with Health Checks</strong></p><pre>wait-for-service-stability: true</pre><ul><li>Starts new tasks with new image</li><li>Waits for health checks to pass</li><li>Only then stops old tasks</li><li>Result: zero downtime deployment</li></ul><h3>Part C: Testing the Pipeline</h3><h3>Step 9: Initial Deployment</h3><p><strong>What we’re doing:</strong> Committing our workflow file and watching the first automated deployment.</p><p><strong>Why this matters:</strong> This is the moment of truth — seeing your entire pipeline work end-to-end for the first time.</p><pre># Make sure you&#39;re in your repository root<br>git status<br><br># Stage the workflow file<br>git add .<br># Commit<br>git commit -m &quot;Add CI/CD pipeline with GitHub Actions&quot;<br># Push to trigger the workflow<br>git push origin main</pre><h3>Step 10: Monitor the Deployment</h3><p><strong>What we’re doing:</strong> Watching the pipeline execute in real-time and verifying each stage.</p><p><strong>Why this matters:</strong> Understanding how to monitor deployments helps you quickly identify and fix issues.</p><p><strong>Via GitHub UI:</strong></p><ol><li>Go to your repository on GitHub</li><li>Click <strong>Actions</strong> tab</li><li>Click on your running workflow</li><li>Watch each job execute in real-time</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*graxczj6LDeaZIhluWfEdw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6D_NwHtgAkeLTnFtqY-2Xg.png" /></figure><p><strong>What you should see:</strong></p><ol><li>✅ Test job completes (linting, tests)</li><li>✅ Security scan completes</li><li>✅ Build job starts (only after tests pass)</li><li>✅ Docker image builds and pushes</li><li>✅ ECS service updates</li><li>✅ Health checks pass</li><li>✅ Old tasks drain and stop</li><li>✅ Deployment summary appears</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--FgCUp_Z-m5jHK5DVk5Fg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjHzoNL53TSbGzDTZXXFlQ.png" /></figure><h3>Step 11: Make a Test Change</h3><p><strong>What we’re doing:</strong> Making a small code change to see the full CI/CD cycle in action.</p><p><strong>Why this matters:</strong> This proves your pipeline is working and gives you confidence to use it daily.</p><p>Create a feature branch:</p><pre># Create a new branch (replace &#39;feature&#39; with your branch name)<br>git branch feature<br><br># Switch to that branch<br>git checkout feature</pre><p>Let’s change the colour of the header</p><p>Navigate to the template folder, open the base.html and change the class of the header to bg-green-600 from bg-navy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2tL8PmGVGhReKbsS3ZECQ.png" /></figure><p>Deploy the change:</p><pre>git add .<br>git commit -m &quot;UI header colour changed to green&quot;<br>git push origin feature</pre><p>Merge the pull the request from feature branch to the main branch and click on Actions to see the pipeline processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2ExauOw_z-fw1ax5C5u7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7zx4Z0iFgz8Pzy_tS1YDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XPAgHLmETSDY6iV5TcWBQA.png" /></figure><p><strong>Watch it deploy automatically!</strong></p><p>Within 3–4 minutes:</p><ul><li>Tests run ✅</li><li>Security scan completes ✅</li><li>New image builds ✅</li><li>ECS updates ✅</li><li>Your change is live! ✅</li></ul><p><strong>Verify the update:</strong></p><pre># Get your ALB URL (from Part 1 terraform output)<br>curl http://your-alb-url.elb.amazonaws.com/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VWAjvK3rhuimmwofirTDOA.png" /></figure><h3>Part D: Advanced Pipeline Features</h3><h3>Step 12: Adding Deployment Rollback</h3><p><strong>What we’re doing:</strong> Creating a workflow to quickly rollback to a previous version if something goes wrong.</p><p><strong>Why this matters:</strong></p><ul><li><strong>Fast Recovery</strong>: Restore service in minutes</li><li><strong>Reduces Stress</strong>: Know you can always undo</li><li><strong>Learning Tool</strong>: Roll back, investigate, fix, redeploy</li><li><strong>Business Continuity</strong>: Minimize downtime</li></ul><p>Create .github/workflows/rollback.yml:</p><pre>name: Rollback Deployment<br><br>on:<br>  workflow_dispatch:<br>    inputs:<br>      task_revision:<br>        description: &#39;Task definition revision to rollback to&#39;<br>        required: true<br>        type: string<br><br>env:<br>  AWS_REGION: eu-west-2<br>  ECS_CLUSTER: attendance-app-dev-cluster<br>  ECS_SERVICE: attendance-app-dev-service<br><br>jobs:<br>  rollback:<br>    name: Rollback to Previous Version<br>    runs-on: ubuntu-latest<br>    permissions:<br>      id-token: write<br>      contents: read<br><br>    steps:<br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br>          aws-region: ${{ env.AWS_REGION }}<br><br>      - name: Rollback to specified revision<br>        run: |<br>          aws ecs update-service \<br>            --cluster ${{ env.ECS_CLUSTER }} \<br>            --service ${{ env.ECS_SERVICE }} \<br>            --task-definition attendance-app-dev:${{ inputs.task_revision }} \<br>            --force-new-deployment<br><br>      - name: Wait for service stability<br>        run: |<br>          aws ecs wait services-stable \<br>            --cluster ${{ env.ECS_CLUSTER }} \<br>            --services ${{ env.ECS_SERVICE }}<br><br>      - name: Rollback Summary<br>        run: |<br>          echo &quot;### ⏪ Rollback Successful&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Rolled back to revision:** ${{ inputs.task_revision }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Cluster:** ${{ env.ECS_CLUSTER }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Service:** ${{ env.ECS_SERVICE }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Triggered by:** @${{ github.actor }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY</pre><p><strong>How to use:</strong></p><ol><li>Go to Actions tab</li><li>Select “Rollback Deployment”</li><li>Click “Run workflow”</li><li>Enter the ECS Task definition revision to rollback to</li><li>Click “Run workflow”</li></ol><p>To get the ECS Task definition that you want to roll back to, check this on the ECS cluster specific to this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZQ57Sd7aXdmaeW_vfuyJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*uaW9zPEtW36LtiuXz4n4QQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-4KU_tGG3apv7NiC1UJ7Sw.png" /></figure><p>We can confirm the previous deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PICvhWtYWs35Y3q7MuQ55Q.png" /></figure><h3>Step 13: Automating Infrastructure Deployment</h3><p><strong>What we’re doing:</strong> Creating a GitHub Actions workflow to automatically apply Terraform infrastructure changes, eliminating the need for manual terraform apply commands.</p><p><strong>Why this matters:</strong></p><ul><li><strong>GitOps Approach</strong>: Infrastructure changes flow through Git just like application code</li><li><strong>Audit Trail</strong>: Every infrastructure change is tracked with commit history</li><li><strong>Consistency</strong>: Same deployment process every time, no human error</li><li><strong>Automation</strong>: Push to main branch = infrastructure updates automatically</li><li><strong>Safety</strong>: Terraform plan runs before apply, showing exactly what will change</li></ul><p>Create .github/workflows/terraform-apply.yml:</p><pre>name: Terraform Apply<br><br>on:<br>  push:<br>    branches:<br>      - main<br>    paths:<br>      - &#39;terraform/**&#39;<br>  workflow_dispatch:<br><br>env:<br>  TF_VERSION: &#39;1.6.0&#39;<br>  AWS_REGION: eu-west-2<br><br>jobs:<br>  terraform-apply:<br>    name: Terraform Apply<br>    runs-on: ubuntu-latest<br>    permissions:<br>      contents: read<br>      id-token: write<br><br>    defaults:<br>      run:<br>        working-directory: ./terraform<br><br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}<br>          aws-region: ${{ env.AWS_REGION }}<br><br>      - name: Setup Terraform<br>        uses: hashicorp/setup-terraform@v3<br>        with:<br>          terraform_version: ${{ env.TF_VERSION }}<br><br>      - name: Terraform Init<br>        run: |<br>          terraform init \<br>            -backend-config=&quot;bucket=${{ secrets.TF_STATE_BUCKET }}&quot; \<br>            -backend-config=&quot;key=attendance-app/terraform.tfstate&quot; \<br>            -backend-config=&quot;region=${{ env.AWS_REGION }}&quot;<br><br>      - name: Terraform Plan<br>        run: |<br>          terraform plan -no-color -input=false \<br>            -var=&quot;environment=dev&quot; \<br>            -var=&quot;project_name=attendance-app&quot; \<br>            -var=&quot;domain_name=tech-with-tobi.com&quot; \<br>            -var=&quot;route53_zone_name=tech-with-tobi.com&quot; \<br>            -var=&quot;create_route53_record=true&quot; \<br>            -var=&quot;db_username=${{ secrets.DB_USERNAME }}&quot; \<br>            -var=&quot;db_password=${{ secrets.DB_PASSWORD }}&quot; \<br>            -out=tfplan<br><br>      - name: Terraform Apply<br>        run: terraform apply -auto-approve tfplan<br><br>      - name: Get Outputs<br>        id: outputs<br>        run: |<br>          echo &quot;alb_dns=$(terraform output -raw alb_dns_name)&quot; &gt;&gt; $GITHUB_OUTPUT<br>          echo &quot;app_url=$(terraform output -raw application_url)&quot; &gt;&gt; $GITHUB_OUTPUT<br><br>      - name: Deployment Summary<br>        run: |<br>          echo &quot;### 🏗️ Infrastructure Deployed Successfully!&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Application URL:** ${{ steps.outputs.outputs.app_url }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**ALB DNS:** ${{ steps.outputs.outputs.alb_dns }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Region:** ${{ env.AWS_REGION }}&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;**Environment:** dev&quot; &gt;&gt; $GITHUB_STEP_SUMMARY</pre><p><strong>Key features:</strong></p><ul><li>Triggers on changes to terraform/** directory</li><li>Uses OIDC for secure AWS authentication</li><li>Stores plan output for review</li><li>Provides deployment summary with infrastructure URLs</li></ul><h4>Workflow 2: Terraform Destroy</h4><p>This workflow safely destroys all infrastructure with confirmation required.</p><p>Create .github/workflows/terraform-destroy.yml:</p><pre>name: Terraform Destroy<br><br>on:<br>  workflow_dispatch:<br>    inputs:<br>      confirmation:<br>        description: &#39;Type &quot;destroy&quot; to confirm deletion of all resources&#39;<br>        required: true<br>        default: &#39;&#39;<br><br>jobs:<br>  destroy:<br>    name: Destroy Infrastructure<br>    runs-on: ubuntu-latest<br>    <br>    if: github.event.inputs.confirmation == &#39;destroy&#39;<br>    <br>    permissions:<br>      id-token: write<br>      contents: read<br>    <br>    defaults:<br>      run:<br>        working-directory: terraform  # &lt;-- ADD THIS LINE<br>    <br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: arn:aws:iam::180048382895:role/GitHubActionsECSRole<br>          aws-region: eu-west-2<br><br>      - name: Setup Terraform<br>        uses: hashicorp/setup-terraform@v3<br>        with:<br>          terraform_version: 1.6.0<br><br>      - name: Terraform Init<br>        run: |<br>          terraform init \<br>            -backend-config=&quot;bucket=attendance-app-s3-backend&quot; \<br>            -backend-config=&quot;key=attendance-app/terraform.tfstate&quot; \<br>            -backend-config=&quot;region=eu-west-2&quot;<br><br>      - name: Terraform Destroy<br>        run: |<br>          terraform destroy -auto-approve \<br>            -var=&quot;environment=dev&quot; \<br>            -var=&quot;project_name=attendance-app&quot; \<br>            -var=&quot;db_username=${{ secrets.DB_USERNAME }}&quot; \<br>            -var=&quot;db_password=${{ secrets.DB_PASSWORD }}&quot;<br><br>      - name: Destroy Complete<br>        run: |<br>          echo &quot;### 🗑️ Infrastructure Destroyed Successfully!&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br>          echo &quot;All AWS resources have been removed.&quot; &gt;&gt; $GITHUB_STEP_SUMMARY<br></pre><pre>### **Additional GitHub Secrets Needed**<br><br>Add this secret to your repository (in addition to the ones from Part A):<br>Name: TF_STATE_BUCKET<br>Value: attendance-app-s3-backend</pre><h3>Testing the Terraform Workflows</h3><p><strong>To test Terraform Apply:</strong></p><pre># Make a change to your Terraform code<br>cd terraform<br># Edit any .tf file<br>git add .<br>git commit -m &quot;Update infrastructure configuration&quot;<br>git push origin main<br><br># Watch the workflow run in Actions tab</pre><p><strong>To test Terraform Destroy:</strong></p><ol><li>Go to <strong>Actions</strong> tab</li><li>Select <strong>“Terraform Destroy”</strong> workflow</li><li>Click <strong>“Run workflow”</strong></li><li>Type destroy in the confirmation box</li><li>Click <strong>“Run workflow”</strong></li></ol><h3>What We’ve Accomplished</h3><p>🎉 <strong>Congratulations!</strong> You now have a production-ready CI/CD pipeline with:</p><p><strong>Part 1:</strong> Built production-grade infrastructure with Terraform</p><ul><li>VPC, subnets, security groups</li><li>Application Load Balancer</li><li>ECS Fargate cluster</li><li>RDS PostgreSQL database</li><li>CloudWatch monitoring</li></ul><p><strong>Part 2 (This Article):</strong> Automated the entire deployment process</p><ul><li>CI/CD pipeline with GitHub Actions</li><li>Automated testing and security scanning</li><li>Zero-downtime deployments</li><li>Rollback capabilities</li></ul><h3>Next Steps</h3><h3>Part 3: Advanced Monitoring and Observability with Cloudwatch Log (Coming Next)</h3><h3>Share Your Success! 🚀</h3><p>If you’ve successfully built this CI/CD pipeline:</p><ul><li>Share your experience on LinkedIn</li><li>Tweet your deployment metrics</li><li>Write about your learnings</li><li>Help others in the comments</li></ul><p><strong>Connect with me</strong> on <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">LinkedIn</a> for more AWS, DevOps, and cloud architecture content. Stay tuned for Part 3 where we’ll add Advanced Monitoring!</p><p><strong>Use these hashtags:</strong> #DevOps #CICD #GitHubActions #AWS #ECS #Docker #InfrastructureAsCode #CloudComputing</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66082cc70f17" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating AWS ECS Deployment — Part 1: Infrastructure as Code with Terraform]]></title>
            <link>https://medium.com/@tobi.akinlade.co/automating-aws-ecs-deployment-part-1-infrastructure-as-code-with-terraform-9ce9e69d6436?source=rss-96a31a8dcfdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ce9e69d6436</guid>
            <dc:creator><![CDATA[Oluwatobi Akinlade]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 19:38:59 GMT</pubDate>
            <atom:updated>2025-10-10T19:38:59.821Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating AWS ECS Deployment — Part 1: Infrastructure as Code with Terraform</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YRTVxN4YnDKFjvh8BdcljA.png" /></figure><p>New to AWS infrastructure? Start with the manual approach first! I recommend checking out my hands-on guide where you’ll build this same architecture using the AWS Console. Once you understand how each component works, coming back to this Terraform tutorial will make much more sense.</p><p>Manual setup guide: <a href="https://medium.com/@tobi.akinlade.co/deploying-a-monolithic-flask-application-on-aws-ecs-with-3-tier-architecture-89a891c72d69">https://medium.com/@tobi.akinlade.co/deploying-a-monolithic-flask-application-on-aws-ecs-with-3-tier-architecture-89a891c72d69</a></p><h3>What We’ll Build in Part 1</h3><p>By the end of this article, you’ll have a <strong>fully functional 3-tier application</strong> running on AWS ECS, provisioned entirely through Terraform. The infrastructure will include:</p><ul><li>✅ VPC with public and private subnets across 2 availability zones</li><li>✅ Application Load Balancer with health checks</li><li>✅ ECS Cluster with Fargate tasks</li><li>✅ RDS PostgreSQL database</li><li>✅ All necessary security groups and networking</li><li>✅ A working Flask attendance application (pre-built, ready to deploy)</li><li>✅ <strong>Optional:</strong> Custom domain with SSL/TLS certificate</li></ul><p><strong>No coding required!</strong> We’ll use a pre-built Docker image for the application, so you can focus on learning the infrastructure.</p><h3>Prerequisites</h3><p>Before starting, ensure you have:</p><ul><li>AWS CLI installed and configured (aws configure)</li><li>Terraform installed (version 1.0+)</li><li>Docker installed locally (to pull and push images)</li><li>Basic understanding of AWS services</li><li>An AWS account with appropriate permissions</li><li><strong>Optional:</strong> A domain name (for SSL setup)</li></ul><p><strong>Note:</strong> You don’t need any application code — we’ll use a pre-built Docker image!</p><h3>Step 1: Get the Application Image</h3><p><strong>What we’re doing:</strong> Getting a working Docker image of the Flask application that we’ll deploy to AWS. Instead of building from source (which can have errors), we’ll pull a pre-tested image from Docker Hub.</p><p><strong>Why this matters:</strong> Using a pre-built image ensures everyone following this tutorial gets the same results without dealing with build errors, dependency issues, or version conflicts.</p><h3>Use Pre-built Image</h3><pre># Pull the pre-built image from Docker Hub<br>docker pull tobiakinlade/attendance-app:latest</pre><h3>Step 2: Set Up Terraform Backend</h3><p><strong>What we’re doing:</strong> Creating a secure, centralised location to store Terraform’s state file. The state file keeps track of all the infrastructure Terraform manages.</p><p><strong>Why this matters:</strong></p><ul><li><strong>State locking</strong> prevents multiple people from making changes simultaneously</li><li><strong>Remote storage</strong> allows team collaboration</li><li><strong>Versioning</strong> enables rollback if needed</li><li><strong>Security</strong> keeps sensitive data encrypted</li></ul><h3>Create S3 Bucket and DynamoDB Table</h3><pre># Create S3 bucket for state (use a unique name)<br>aws s3 mb s3://your-terraform-state-bucket-unique-name --region eu-west-2<br><br># Enable versioning<br>aws s3api put-bucket-versioning \<br>  --bucket your-terraform-state-bucket-unique-name \<br>  --versioning-configuration Status=Enabled<br><br># Enable encryption<br>aws s3api put-bucket-encryption \<br>  --bucket your-terraform-state-bucket-unique-name \<br>  --server-side-encryption-configuration &#39;{<br>    &quot;Rules&quot;: [{<br>      &quot;ApplyServerSideEncryptionByDefault&quot;: {<br>        &quot;SSEAlgorithm&quot;: &quot;AES256&quot;<br>      }<br>    }]<br>  }&#39;<br><br># Create DynamoDB table for state locking<br>aws dynamodb create-table \<br>  --table-name terraform-state-lock \<br>  --attribute-definitions AttributeName=LockID,AttributeType=S \<br>  --key-schema AttributeName=LockID,KeyType=HASH \<br>  --billing-mode PAY_PER_REQUEST \<br>  --region eu-west-2</pre><h3>Backend Configuration</h3><p><strong>terraform/backend.tf:</strong></p><pre>terraform {<br>  required_version = &quot;&gt;= 1.0&quot;<br>  <br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 5.0&quot;<br>    }<br>  }<br><br>  backend &quot;s3&quot; {<br>    bucket         = &quot;attendance-app-s3-backend&quot;<br>    key            = &quot;ecs-app/terraform.tfstate&quot;<br>    region         = &quot;eu-west-2&quot;<br>    encrypt        = true<br>    dynamodb_table = &quot;attendance-app-state-lock&quot;<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = var.aws_region<br><br>  default_tags {<br>    tags = {<br>      Project     = var.project_name<br>      Environment = var.environment<br>      ManagedBy   = &quot;Terraform&quot;<br>    }<br>  }<br>}</pre><h3>Step 3: Define Variables</h3><p><strong>What we’re doing:</strong> Creating reusable configuration values that make our Terraform code flexible and maintainable. Variables allow us to easily change settings like region, database passwords, or environment names without editing the core infrastructure code.</p><p><strong>Why this matters:</strong></p><ul><li>Deploy the same infrastructure to different environments (dev, staging, prod)</li><li>Keep sensitive data (passwords) separate from code</li><li>Make infrastructure reusable and customizable</li><li>Follow Infrastructure as Code best practices</li></ul><p><strong>terraform/variables.tf:</strong></p><pre>variable &quot;aws_region&quot; {<br>  description = &quot;AWS region&quot;<br>  type        = string<br>  default     = &quot;eu-west-2&quot;<br>}<br><br>variable &quot;project_name&quot; {<br>  description = &quot;Project name used for resource naming&quot;<br>  type        = string<br>  default     = &quot;attendance-app&quot;<br>}<br><br>variable &quot;environment&quot; {<br>  description = &quot;Environment name (dev, staging, prod)&quot;<br>  type        = string<br>  default     = &quot;dev&quot;<br>}<br><br>variable &quot;vpc_cidr&quot; {<br>  description = &quot;CIDR block for VPC&quot;<br>  type        = string<br>  default     = &quot;10.0.0.0/16&quot;<br>}<br><br>variable &quot;availability_zones&quot; {<br>  description = &quot;List of availability zones&quot;<br>  type        = list(string)<br>  default     = [&quot;eu-west-2a&quot;, &quot;eu-west-2b&quot;]<br>}<br><br>variable &quot;container_port&quot; {<br>  description = &quot;Port exposed by the container&quot;<br>  type        = number<br>  default     = 8000<br>}<br><br>variable &quot;db_name&quot; {<br>  description = &quot;Database name&quot;<br>  type        = string<br>  default     = &quot;attendancedb&quot;<br>}<br><br>variable &quot;db_username&quot; {<br>  description = &quot;Database master username&quot;<br>  type        = string<br>  sensitive   = true<br>}<br><br>variable &quot;db_password&quot; {<br>  description = &quot;Database master password&quot;<br>  type        = string<br>  sensitive   = true<br>}<br><br>variable &quot;ecs_task_cpu&quot; {<br>  description = &quot;Fargate task CPU units&quot;<br>  type        = string<br>  default     = &quot;256&quot;<br>}<br><br>variable &quot;ecs_task_memory&quot; {<br>  description = &quot;Fargate task memory in MB&quot;<br>  type        = string<br>  default     = &quot;512&quot;<br>}<br><br>variable &quot;desired_count&quot; {<br>  description = &quot;Number of ECS tasks to run&quot;<br>  type        = number<br>  default     = 2<br>}<br><br>variable &quot;health_check_path&quot; {<br>  description = &quot;Health check endpoint&quot;<br>  type        = string<br>  default     = &quot;/login&quot;<br>}<br><br>variable &quot;domain_name&quot; {<br>  description = &quot;Domain name for the application (tech-with-tobi.com)&quot;<br>  type        = string<br>  default     = &quot;tech-with-tobi.com&quot;<br>}<br><br>variable &quot;route53_zone_name&quot; {<br>  description = &quot;Route 53 hosted zone name (tech-with-tobi.com)&quot;<br>  type        = string<br>  default     = &quot;tech-with-tobi.com&quot;<br>}<br><br>variable &quot;create_route53_record&quot; {<br>  description = &quot;Whether to create Route 53 DNS record&quot;<br>  type        = bool<br>  default     = false<br>}</pre><p><strong>terraform/terraform.tfvars:</strong></p><pre>aws_region     = &quot;eu-west-2&quot;<br>project_name   = &quot;attendance-app&quot;<br>environment    = &quot;dev&quot;<br>vpc_cidr       = &quot;10.0.0.0/16&quot;<br>db_name        = &quot;studentdb&quot;<br>db_username    = &quot;dbadmin&quot;<br>db_password    = &quot;ChangeMe123!SecurePassword&quot;<br>desired_count  = 2<br><br># Optional: Leave empty for HTTP-only deployment<br>domain_name           = &quot;&quot;<br>route53_zone_name     = &quot;&quot;<br>create_route53_record = false<br><br># To enable custom domain, uncomment and fill:<br># domain_name           = &quot;yourdomain.com&quot;<br># route53_zone_name     = &quot;yourdomain.com&quot;<br># create_route53_record = true</pre><h3>Step 4: VPC and Networking</h3><p><strong>What we’re doing:</strong> Building the foundation network infrastructure — a Virtual Private Cloud (VPC) with public and private subnets across two availability zones.</p><p><strong>Why this matters:</strong></p><ul><li><strong>Isolation:</strong> Your application runs in its own private network</li><li><strong>Security:</strong> Private subnets keep your application and database hidden from the internet</li><li><strong>High Availability:</strong> Multiple availability zones ensure your app stays online if one zone fails</li><li><strong>Internet Access:</strong> NAT Gateway allows private resources to download updates while staying secure</li></ul><p><strong>Architecture overview:</strong></p><ul><li><strong>Public subnets</strong> → ALB (faces internet)</li><li><strong>Private app subnets</strong> → ECS tasks (hidden from internet)</li><li><strong>Private DB subnets</strong> → RDS database (most secure layer)</li></ul><p><strong>terraform/vpc.tf:</strong></p><pre># VPC<br>resource &quot;aws_vpc&quot; &quot;main&quot; {<br>  cidr_block           = var.vpc_cidr<br>  enable_dns_hostnames = true<br>  enable_dns_support   = true<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-vpc&quot;<br>  }<br>}<br># Internet Gateway<br>resource &quot;aws_internet_gateway&quot; &quot;main&quot; {<br>  vpc_id = aws_vpc.main.id<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-igw&quot;<br>  }<br>}<br># Public Subnets<br>resource &quot;aws_subnet&quot; &quot;public&quot; {<br>  count                   = 2<br>  vpc_id                  = aws_vpc.main.id<br>  cidr_block              = cidrsubnet(var.vpc_cidr, 8, count.index)<br>  availability_zone       = var.availability_zones[count.index]<br>  map_public_ip_on_launch = true<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-public-${count.index + 1}&quot;<br>    Type = &quot;Public&quot;<br>  }<br>}<br># Private Subnets for ECS<br>resource &quot;aws_subnet&quot; &quot;private_app&quot; {<br>  count             = 2<br>  vpc_id            = aws_vpc.main.id<br>  cidr_block        = cidrsubnet(var.vpc_cidr, 8, count.index + 10)<br>  availability_zone = var.availability_zones[count.index]<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-private-app-${count.index + 1}&quot;<br>    Type = &quot;Private-App&quot;<br>  }<br>}<br># Private Subnets for RDS<br>resource &quot;aws_subnet&quot; &quot;private_db&quot; {<br>  count             = 2<br>  vpc_id            = aws_vpc.main.id<br>  cidr_block        = cidrsubnet(var.vpc_cidr, 8, count.index + 20)<br>  availability_zone = var.availability_zones[count.index]<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-private-db-${count.index + 1}&quot;<br>    Type = &quot;Private-DB&quot;<br>  }<br>}<br># Elastic IP for NAT Gateway<br>resource &quot;aws_eip&quot; &quot;nat&quot; {<br>  domain = &quot;vpc&quot;<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-nat-eip&quot;<br>  }<br>  depends_on = [aws_internet_gateway.main]<br>}<br># NAT Gateway<br>resource &quot;aws_nat_gateway&quot; &quot;main&quot; {<br>  allocation_id = aws_eip.nat.id<br>  subnet_id     = aws_subnet.public[0].id<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-nat&quot;<br>  }<br>  depends_on = [aws_internet_gateway.main]<br>}<br># Public Route Table<br>resource &quot;aws_route_table&quot; &quot;public&quot; {<br>  vpc_id = aws_vpc.main.id<br>  route {<br>    cidr_block = &quot;0.0.0.0/0&quot;<br>    gateway_id = aws_internet_gateway.main.id<br>  }<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-public-rt&quot;<br>  }<br>}<br># Private Route Table<br>resource &quot;aws_route_table&quot; &quot;private&quot; {<br>  vpc_id = aws_vpc.main.id<br>  route {<br>    cidr_block     = &quot;0.0.0.0/0&quot;<br>    nat_gateway_id = aws_nat_gateway.main.id<br>  }<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-private-rt&quot;<br>  }<br>}<br># Route Table Associations<br>resource &quot;aws_route_table_association&quot; &quot;public&quot; {<br>  count          = 2<br>  subnet_id      = aws_subnet.public[count.index].id<br>  route_table_id = aws_route_table.public.id<br>}<br>resource &quot;aws_route_table_association&quot; &quot;private_app&quot; {<br>  count          = 2<br>  subnet_id      = aws_subnet.private_app[count.index].id<br>  route_table_id = aws_route_table.private.id<br>}<br>resource &quot;aws_route_table_association&quot; &quot;private_db&quot; {<br>  count          = 2<br>  subnet_id      = aws_subnet.private_db[count.index].id<br>  route_table_id = aws_route_table.private.id<br>}</pre><h3>Step 5: Security Groups</h3><p><strong>What we’re doing:</strong> Creating virtual firewalls that control exactly what traffic can reach each layer of our application. This implements “defense in depth” — multiple layers of security.</p><p><strong>Why this matters:</strong></p><ul><li><strong>ALB Security Group:</strong> Only allows HTTP/HTTPS from the internet</li><li><strong>ECS Security Group:</strong> Only accepts traffic from the ALB (not direct internet access)</li><li><strong>RDS Security Group:</strong> Only allows database connections from ECS tasks</li><li>Each layer can only talk to what it needs — nothing more</li></ul><p><strong>Security principle:</strong> If an attacker compromises one layer, they can’t automatically access everything else.</p><p><strong>terraform/security-groups.tf:</strong></p><pre># ALB Security Group<br>resource &quot;aws_security_group&quot; &quot;alb&quot; {<br>  name_prefix = &quot;${var.project_name}-${var.environment}-alb-&quot;<br>  description = &quot;Security group for Application Load Balancer&quot;<br>  vpc_id      = aws_vpc.main.id<br>  ingress {<br>    description = &quot;HTTP from Internet&quot;<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  ingress {<br>    description = &quot;HTTPS from Internet&quot;<br>    from_port   = 443<br>    to_port     = 443<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  egress {<br>    description = &quot;Allow all outbound traffic&quot;<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-alb-sg&quot;<br>  }<br>  lifecycle {<br>    create_before_destroy = true<br>  }<br>}<br># ECS Tasks Security Group<br>resource &quot;aws_security_group&quot; &quot;ecs_tasks&quot; {<br>  name_prefix = &quot;${var.project_name}-${var.environment}-ecs-&quot;<br>  description = &quot;Security group for ECS tasks&quot;<br>  vpc_id      = aws_vpc.main.id<br>  ingress {<br>    description     = &quot;Allow inbound from ALB only&quot;<br>    from_port       = var.container_port<br>    to_port         = var.container_port<br>    protocol        = &quot;tcp&quot;<br>    security_groups = [aws_security_group.alb.id]<br>  }<br>  egress {<br>    description = &quot;Allow all outbound traffic&quot;<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-ecs-sg&quot;<br>  }<br>  lifecycle {<br>    create_before_destroy = true<br>  }<br>}<br># RDS Security Group<br>resource &quot;aws_security_group&quot; &quot;rds&quot; {<br>  name_prefix = &quot;${var.project_name}-${var.environment}-rds-&quot;<br>  description = &quot;Security group for RDS database&quot;<br>  vpc_id      = aws_vpc.main.id<br>  ingress {<br>    description     = &quot;PostgreSQL from ECS tasks only&quot;<br>    from_port       = 5432<br>    to_port         = 5432<br>    protocol        = &quot;tcp&quot;<br>    security_groups = [aws_security_group.ecs_tasks.id]<br>  }<br>  egress {<br>    description = &quot;Allow all outbound traffic&quot;<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-rds-sg&quot;<br>  }<br>  lifecycle {<br>    create_before_destroy = true<br>  }<br>}</pre><h3>Step 6: Application Load Balancer</h3><p><strong>What we’re doing:</strong> Setting up an Application Load Balancer (ALB) that distributes incoming traffic across multiple ECS tasks and performs health checks.</p><p><strong>Why this matters:</strong></p><ul><li><strong>High Availability:</strong> If one task fails, traffic goes to healthy tasks</li><li><strong>Scalability:</strong> Can distribute load across many tasks</li><li><strong>Health Checks:</strong> Automatically removes unhealthy tasks from rotation</li><li><strong>SSL Termination:</strong> Handles HTTPS (if configured) so your app doesn’t have to</li><li><strong>Single Entry Point:</strong> One URL for users, many tasks behind it</li></ul><p><strong>How it works:</strong></p><ol><li>User hits ALB DNS/domain</li><li>ALB checks which tasks are healthy</li><li>Forwards request to a healthy task</li><li>Returns response to user</li></ol><p><strong>terraform/alb.tf:</strong></p><pre># Application Load Balancer<br>resource &quot;aws_lb&quot; &quot;main&quot; {<br>  name               = &quot;${var.project_name}-${var.environment}-alb&quot;<br>  internal           = false<br>  load_balancer_type = &quot;application&quot;<br>  security_groups    = [aws_security_group.alb.id]<br>  subnets            = aws_subnet.public[*].id<br><br>  enable_deletion_protection = false<br>  enable_http2              = true<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-alb&quot;<br>  }<br>}<br><br># Target Group<br>resource &quot;aws_lb_target_group&quot; &quot;main&quot; {<br>  name        = &quot;${var.project_name}-${var.environment}-tg&quot;<br>  port        = var.container_port<br>  protocol    = &quot;HTTP&quot;<br>  vpc_id      = aws_vpc.main.id<br>  target_type = &quot;ip&quot;<br><br>  health_check {<br>    enabled             = true<br>    path                = var.health_check_path<br>    port                = &quot;traffic-port&quot;<br>    protocol            = &quot;HTTP&quot;<br>    healthy_threshold   = 2<br>    unhealthy_threshold = 3<br>    timeout             = 5<br>    interval            = 30<br>    matcher             = &quot;200-299&quot;<br>  }<br><br>  deregistration_delay = 30<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-tg&quot;<br>  }<br>}<br><br># HTTP Listener - Redirect to HTTPS if certificate exists<br>resource &quot;aws_lb_listener&quot; &quot;http&quot; {<br>  load_balancer_arn = aws_lb.main.arn<br>  port              = &quot;80&quot;<br>  protocol          = &quot;HTTP&quot;<br><br>  default_action {<br>    type = var.domain_name != &quot;&quot; ? &quot;redirect&quot; : &quot;forward&quot;<br><br>    # Redirect to HTTPS if certificate exists<br>    dynamic &quot;redirect&quot; {<br>      for_each = var.domain_name != &quot;&quot; ? [1] : []<br>      content {<br>        port        = &quot;443&quot;<br>        protocol    = &quot;HTTPS&quot;<br>        status_code = &quot;HTTP_301&quot;<br>      }<br>    }<br><br>    # Forward to target group if no certificate<br>    target_group_arn = var.domain_name == &quot;&quot; ? aws_lb_target_group.main.arn : null<br>  }<br>}<br><br># HTTPS Listener (only if certificate exists)<br>resource &quot;aws_lb_listener&quot; &quot;https&quot; {<br>  count             = var.domain_name != &quot;&quot; ? 1 : 0<br>  load_balancer_arn = aws_lb.main.arn<br>  port              = &quot;443&quot;<br>  protocol          = &quot;HTTPS&quot;<br>  ssl_policy        = &quot;ELBSecurityPolicy-TLS13-1-2-2021-06&quot;<br>  certificate_arn   = aws_acm_certificate.main[0].arn<br><br>  default_action {<br>    type             = &quot;forward&quot;<br>    target_group_arn = aws_lb_target_group.main.arn<br>  }<br><br>  depends_on = [aws_acm_certificate_validation.main]<br>}</pre><h3>Step 7: RDS PostgreSQL Database</h3><p><strong>What we’re doing:</strong> Creating a managed PostgreSQL database that will store your application data securely in private subnets.</p><p><strong>Why this matters:</strong></p><ul><li><strong>Managed Service:</strong> AWS handles backups, patches, and maintenance</li><li><strong>Multi-AZ Option:</strong> Can automatically failover to standby in another zone</li><li><strong>Encrypted:</strong> Data is encrypted at rest</li><li><strong>Isolated:</strong> Lives in private subnets, only accessible from ECS tasks</li><li><strong>Automated Backups:</strong> 7-day retention for disaster recovery</li></ul><p><strong>Key features:</strong></p><ul><li>Daily automated backups</li><li>Point-in-time recovery</li><li>Monitoring via CloudWatch</li><li>Secure connection from application only</li></ul><p><strong>terraform/rds.tf:</strong></p><pre># DB Subnet Group<br>resource &quot;aws_db_subnet_group&quot; &quot;main&quot; {<br>  name       = &quot;${var.project_name}-${var.environment}-db-subnet&quot;<br>  subnet_ids = aws_subnet.private_db[*].id<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-db-subnet&quot;<br>  }<br>}<br><br># RDS Instance<br>resource &quot;aws_db_instance&quot; &quot;main&quot; {<br>  identifier     = &quot;${var.project_name}-${var.environment}-db&quot;<br>  engine         = &quot;postgres&quot;<br>  engine_version = &quot;15&quot;<br>  instance_class = &quot;db.t3.micro&quot;<br><br>  allocated_storage     = 20<br>  max_allocated_storage = 100<br>  storage_type          = &quot;gp3&quot;<br>  storage_encrypted     = true<br><br>  db_name  = var.db_name<br>  username = var.db_username<br>  password = var.db_password<br><br>  db_subnet_group_name   = aws_db_subnet_group.main.name<br>  vpc_security_group_ids = [aws_security_group.rds.id]<br>  publicly_accessible    = false<br><br>  backup_retention_period = 7<br>  backup_window          = &quot;03:00-04:00&quot;<br>  maintenance_window     = &quot;sun:04:00-sun:05:00&quot;<br><br>  enabled_cloudwatch_logs_exports = [&quot;postgresql&quot;, &quot;upgrade&quot;]<br>  <br>  skip_final_snapshot       = true<br>  final_snapshot_identifier = &quot;${var.project_name}-${var.environment}-final-snapshot-${formatdate(&quot;YYYYMMDDhhmmss&quot;, timestamp())}&quot;<br><br>  deletion_protection = false<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-db&quot;<br>  }<br><br>  lifecycle {<br>    ignore_changes = [final_snapshot_identifier]<br>  }<br>}</pre><h3>Step 8: ECS Cluster and Service</h3><p><strong>What we’re doing:</strong> Creating the core compute layer — an ECS cluster that runs your containerised application using AWS Fargate (serverless containers).</p><p><strong>Why this matters:</strong></p><ul><li><strong>No Server Management:</strong> Fargate runs containers without managing EC2 instances</li><li><strong>Auto Scaling:</strong> Can automatically add/remove tasks based on load</li><li><strong>Container Insights:</strong> Built-in monitoring and logging</li><li><strong>High Availability:</strong> Runs multiple tasks across availability zones</li><li><strong>Easy Updates:</strong> Rolling deployments with zero downtime</li></ul><p><strong>Components created:</strong></p><ol><li><strong>ECR Repository</strong> — Stores your Docker images (with automatic scanning)</li><li><strong>ECS Cluster</strong> — Logical grouping of tasks</li><li><strong>Task Definition</strong> — Blueprint for your container (CPU, memory, environment variables)</li><li><strong>ECS Service</strong> — Maintains desired number of running tasks</li><li><strong>IAM Roles</strong> — Permissions for tasks to access AWS services</li></ol><p><strong>terraform/ecs.tf:</strong></p><pre># ECR Repository<br>resource &quot;aws_ecr_repository&quot; &quot;main&quot; {<br>  name                 = &quot;${var.project_name}-${var.environment}&quot;<br>  image_tag_mutability = &quot;MUTABLE&quot;<br>  force_delete = true<br><br>  image_scanning_configuration {<br>    scan_on_push = true<br>  }<br><br>  encryption_configuration {<br>    encryption_type = &quot;AES256&quot;<br>  }<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}&quot;<br>  }<br>}<br><br># ECR Lifecycle Policy<br>resource &quot;aws_ecr_lifecycle_policy&quot; &quot;main&quot; {<br>  repository = aws_ecr_repository.main.name<br><br>  policy = jsonencode({<br>    rules = [{<br>      rulePriority = 1<br>      description  = &quot;Keep last 10 images&quot;<br>      selection = {<br>        tagStatus     = &quot;any&quot;<br>        countType     = &quot;imageCountMoreThan&quot;<br>        countNumber   = 10<br>      }<br>      action = {<br>        type = &quot;expire&quot;<br>      }<br>    }]<br>  })<br>}<br><br># ECS Cluster<br>resource &quot;aws_ecs_cluster&quot; &quot;main&quot; {<br>  name = &quot;${var.project_name}-${var.environment}-cluster&quot;<br><br>  setting {<br>    name  = &quot;containerInsights&quot;<br>    value = &quot;enabled&quot;<br>  }<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-cluster&quot;<br>  }<br>}<br><br># CloudWatch Log Group<br>resource &quot;aws_cloudwatch_log_group&quot; &quot;ecs&quot; {<br>  name              = &quot;/ecs/${var.project_name}-${var.environment}&quot;<br>  retention_in_days = 7<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-logs&quot;<br>  }<br>}<br><br># ECS Task Execution Role<br>resource &quot;aws_iam_role&quot; &quot;ecs_task_execution&quot; {<br>  name = &quot;${var.project_name}-${var.environment}-ecs-exec-role&quot;<br><br>  assume_role_policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [{<br>      Action = &quot;sts:AssumeRole&quot;<br>      Effect = &quot;Allow&quot;<br>      Principal = {<br>        Service = &quot;ecs-tasks.amazonaws.com&quot;<br>      }<br>    }]<br>  })<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-ecs-exec-role&quot;<br>  }<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;ecs_task_execution&quot; {<br>  role       = aws_iam_role.ecs_task_execution.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy&quot;<br>}<br><br># ECS Task Role (for application permissions)<br>resource &quot;aws_iam_role&quot; &quot;ecs_task&quot; {<br>  name = &quot;${var.project_name}-${var.environment}-ecs-task-role&quot;<br><br>  assume_role_policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [{<br>      Action = &quot;sts:AssumeRole&quot;<br>      Effect = &quot;Allow&quot;<br>      Principal = {<br>        Service = &quot;ecs-tasks.amazonaws.com&quot;<br>      }<br>    }]<br>  })<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-ecs-task-role&quot;<br>  }<br>}<br><br># Task Definition<br>resource &quot;aws_ecs_task_definition&quot; &quot;main&quot; {<br>  family                   = &quot;${var.project_name}-${var.environment}&quot;<br>  network_mode             = &quot;awsvpc&quot;<br>  requires_compatibilities = [&quot;FARGATE&quot;]<br>  cpu                      = var.ecs_task_cpu<br>  memory                   = var.ecs_task_memory<br>  execution_role_arn       = aws_iam_role.ecs_task_execution.arn<br>  task_role_arn            = aws_iam_role.ecs_task.arn<br><br>  container_definitions = jsonencode([{<br>    name      = &quot;${var.project_name}-container&quot;<br>    image     = &quot;${aws_ecr_repository.main.repository_url}:latest&quot;<br>    essential = true<br><br>    portMappings = [{<br>      containerPort = var.container_port<br>      protocol      = &quot;tcp&quot;<br>    }]<br><br>    environment = [<br>      {<br>        name  = &quot;DB_LINK&quot;<br>        value = &quot;postgresql://${var.db_username}:${var.db_password}@${aws_db_instance.main.endpoint}/${var.db_name}&quot;<br>      },<br>      {<br>        name  = &quot;ENVIRONMENT&quot;<br>        value = var.environment<br>      }<br>    ]<br><br>    logConfiguration = {<br>      logDriver = &quot;awslogs&quot;<br>      options = {<br>        &quot;awslogs-group&quot;         = aws_cloudwatch_log_group.ecs.name<br>        &quot;awslogs-region&quot;        = var.aws_region<br>        &quot;awslogs-stream-prefix&quot; = &quot;ecs&quot;<br>      }<br>    }<br><br>    healthCheck = {<br>      command     = [&quot;CMD-SHELL&quot;, &quot;curl -f http://localhost:${var.container_port}${var.health_check_path} || exit 1&quot;]<br>      interval    = 30<br>      timeout     = 5<br>      retries     = 3<br>      startPeriod = 60<br>    }<br>  }])<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-task&quot;<br>  }<br>}<br><br># ECS Service<br>resource &quot;aws_ecs_service&quot; &quot;main&quot; {<br>  name            = &quot;${var.project_name}-${var.environment}-service&quot;<br>  cluster         = aws_ecs_cluster.main.id<br>  task_definition = aws_ecs_task_definition.main.arn<br>  desired_count   = var.desired_count<br>  launch_type     = &quot;FARGATE&quot;<br><br>  network_configuration {<br>    subnets          = aws_subnet.private_app[*].id<br>    security_groups  = [aws_security_group.ecs_tasks.id]<br>    assign_public_ip = false<br>  }<br><br>  load_balancer {<br>    target_group_arn = aws_lb_target_group.main.arn<br>    container_name   = &quot;${var.project_name}-container&quot;<br>    container_port   = var.container_port<br>  }<br><br>  health_check_grace_period_seconds = 300<br><br><br>  depends_on = [<br>    aws_lb_listener.http,<br>    aws_iam_role_policy_attachment.ecs_task_execution<br>  ]<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-service&quot;<br>  }<br>}</pre><h3>Step 9: Optional — SSL Certificate and Custom Domain</h3><p><strong>What we’re doing:</strong> Adding a custom domain name and SSL/TLS certificate for HTTPS access. This section is completely optional — your application works fine with just the ALB DNS name and HTTP.</p><p><strong>Why add this (optional):</strong></p><ul><li><strong>Professional appearance:</strong> app.yourdomain.com vs alb-123456.eu-west-2.elb.amazonaws.com</li><li><strong>Security:</strong> HTTPS encrypts data in transit</li><li><strong>Trust:</strong> Browser shows padlock, no security warnings</li><li><strong>SEO:</strong> Search engines prefer HTTPS sites</li><li><strong>Free SSL:</strong> AWS Certificate Manager provides free certificates with auto-renewal</li></ul><p><strong>Skip this if:</strong></p><ul><li>You’re just learning/testing</li><li>You don’t own a domain</li><li>HTTP is sufficient for your needs</li></ul><p><strong>terraform/acm.tf:</strong></p><pre># Data source to get the Route 53 hosted zone<br>data &quot;aws_route53_zone&quot; &quot;main&quot; {<br>  count = var.create_route53_record ? 1 : 0<br>  name  = var.route53_zone_name<br>}<br><br># Request ACM certificate<br>resource &quot;aws_acm_certificate&quot; &quot;main&quot; {<br>  count             = var.domain_name != &quot;&quot; ? 1 : 0<br>  domain_name       = var.domain_name<br>  validation_method = &quot;DNS&quot;<br><br>  lifecycle {<br>    create_before_destroy = true<br>  }<br><br>  tags = {<br>    Name = &quot;${var.project_name}-${var.environment}-cert&quot;<br>  }<br>}<br><br># Create DNS record for certificate validation<br>resource &quot;aws_route53_record&quot; &quot;cert_validation&quot; {<br>  for_each = var.create_route53_record ? {<br>    for dvo in aws_acm_certificate.main[0].domain_validation_options : dvo.domain_name =&gt; {<br>      name   = dvo.resource_record_name<br>      record = dvo.resource_record_value<br>      type   = dvo.resource_record_type<br>    }<br>  } : {}<br><br>  allow_overwrite = true<br>  name            = each.value.name<br>  records         = [each.value.record]<br>  ttl             = 60<br>  type            = each.value.type<br>  zone_id         = data.aws_route53_zone.main[0].zone_id<br>}<br><br># Wait for certificate validation<br>resource &quot;aws_acm_certificate_validation&quot; &quot;main&quot; {<br>  count                   = var.domain_name != &quot;&quot; ? 1 : 0<br>  certificate_arn         = aws_acm_certificate.main[0].arn<br>  validation_record_fqdns = var.create_route53_record ? [for record in aws_route53_record.cert_validation : record.fqdn] : []<br><br>  timeouts {<br>    create = &quot;10m&quot;<br>  }<br>}</pre><p><strong>terraform/route53.tf:</strong></p><pre># Create A record pointing to ALB<br>resource &quot;aws_route53_record&quot; &quot;main&quot; {<br>  count   = var.create_route53_record ? 1 : 0<br>  zone_id = data.aws_route53_zone.main[0].zone_id<br>  name    = var.domain_name<br>  type    = &quot;A&quot;<br><br>  alias {<br>    name                   = aws_lb.main.dns_name<br>    zone_id                = aws_lb.main.zone_id<br>    evaluate_target_health = true<br>  }<br>}</pre><h3>Step 10: Outputs</h3><p><strong>terraform/outputs.tf:</strong></p><pre>output &quot;vpc_id&quot; {<br>  description = &quot;VPC ID&quot;<br>  value       = aws_vpc.main.id<br>}<br><br>output &quot;alb_dns_name&quot; {<br>  description = &quot;Load Balancer DNS name&quot;<br>  value       = aws_lb.main.dns_name<br>}<br><br>output &quot;alb_url&quot; {<br>  description = &quot;Application URL&quot;<br>  value       = &quot;http://${aws_lb.main.dns_name}&quot;<br>}<br><br>output &quot;ecr_repository_url&quot; {<br>  description = &quot;ECR repository URL&quot;<br>  value       = aws_ecr_repository.main.repository_url<br>}<br><br>output &quot;ecs_cluster_name&quot; {<br>  description = &quot;ECS cluster name&quot;<br>  value       = aws_ecs_cluster.main.name<br>}<br><br>output &quot;ecs_service_name&quot; {<br>  description = &quot;ECS service name&quot;<br>  value       = aws_ecs_service.main.name<br>}<br><br>output &quot;rds_endpoint&quot; {<br>  description = &quot;RDS endpoint&quot;<br>  value       = aws_db_instance.main.endpoint<br>  sensitive   = true<br>}<br><br>output &quot;ecr_push_commands&quot; {<br>  description = &quot;Commands to push Docker image to ECR&quot;<br>  value = &lt;&lt;-EOT<br>    # Login to ECR<br>    aws ecr get-login-password --region ${var.aws_region} | docker login --username AWS --password-stdin ${aws_ecr_repository.main.repository_url}<br>    <br>    # Tag your image<br>    docker tag attendance-app:latest ${aws_ecr_repository.main.repository_url}:latest<br>    <br>    # Push to ECR<br>    docker push ${aws_ecr_repository.main.repository_url}:latest<br>  EOT<br>}<br><br>output &quot;application_url&quot; {<br>  description = &quot;Application URL (custom domain or ALB DNS)&quot;<br>  value       = var.domain_name != &quot;&quot; ? &quot;https://${var.domain_name}&quot; : &quot;http://${aws_lb.main.dns_name}&quot;<br>}<br><br>output &quot;certificate_arn&quot; {<br>  description = &quot;ACM certificate ARN&quot;<br>  value       = var.domain_name != &quot;&quot; ? aws_acm_certificate.main[0].arn : &quot;N/A - No certificate created&quot;<br>}<br><br>output &quot;route53_record&quot; {<br>  description = &quot;Route 53 DNS record&quot;<br>  value       = var.create_route53_record ? aws_route53_record.main[0].fqdn : &quot;N/A - No Route 53 record created&quot;<br>}</pre><p><strong>terraform/main.tf:</strong></p><pre># Data sources<br>data &quot;aws_caller_identity&quot; &quot;current&quot; {}<br><br>data &quot;aws_region&quot; &quot;current&quot; {}<br><br># Local values<br>locals {<br>  account_id = data.aws_caller_identity.current.account_id<br>  region     = data.aws_region.current.name<br>  <br>  common_tags = {<br>    Project     = var.project_name<br>    Environment = var.environment<br>    ManagedBy   = &quot;Terraform&quot;<br>  }<br>}</pre><h3>Step 11: Deploy the Infrastructure</h3><h3>Initialize Terraform</h3><pre>cd terraform<br>terraform init</pre><p>You should see:</p><pre>Terraform has been successfully initialized!</pre><h3>Validate Configuration</h3><pre>terraform validate</pre><p>Expected output:</p><pre>Success! The configuration is valid.</pre><h3>Plan the Deployment</h3><pre>terraform plan</pre><p>Review the plan. Terraform will show approximately 40+ resources to be created.</p><h3>Apply the Configuration</h3><pre>terraform apply --auto-approve</pre><p>This will take <strong>10-15 minutes</strong>.</p><p>If no issue, you should have an output similar to this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HCNhNoS8_pwNNAIfhec7hw.png" /></figure><h3>Step 12: Push Docker Image to ECR</h3><p>After terraform apply completes, we&#39;ll push the application image to ECR.</p><h3>Pull from Docker Hub and Push to ECR</h3><pre># Get the ECR repository URL<br>ECR_URL=$(terraform output -raw ecr_repository_url)<br>echo &quot;ECR URL: $ECR_URL&quot;<br># Go back to project root (if you&#39;re in terraform directory)<br>cd ..<br># Pull the pre-built image from Docker Hub<br>docker pull tobiakinlade/attendance-app:latest<br># Login to ECR<br>aws ecr get-login-password --region eu-west-2 | \<br>  docker login --username AWS --password-stdin $ECR_URL<br># Tag the Docker Hub image for ECR<br>docker tag tobiakinlade/attendance-app:latest $ECR_URL<br># Push to ECR<br>docker push $ECR_URL</pre><p><strong>Expected output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X22iw1CO3_8bxfh7L8Qnbw.png" /></figure><h3>Step 13: Deploy to ECS</h3><p>Force ECS to pull and deploy the new image:</p><pre>aws ecs update-service \<br>  --cluster attendance-app-dev-cluster \<br>  --service attendance-app-dev-service \<br>  --force-new-deployment \<br>  --region eu-west-2</pre><h3>Step 14: Verify Your Application</h3><h3>Monitor Deployment</h3><pre># Check service status (run every 30 seconds)<br>aws ecs describe-services \<br>  --cluster attendance-app-dev-cluster \<br>  --services attendance-app-dev-service \<br>  --query &#39;services[0].{Status:status,Running:runningCount,Desired:desiredCount}&#39; \<br>  --region eu-west-2</pre><p>Wait until Running: 2 and Desired: 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_KemfqwK-LFBpyykFdQJA.png" /></figure><h3>View Logs</h3><pre>aws logs tail /ecs/attendance-app-dev --follow --region eu-west-2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_IBCqkOa8U2CqFbIdHYgA.png" /></figure><h3>Access Your Application</h3><pre>cd terraform<br>terraform output application_url</pre><p>Open the URL in your browser. You should see your Attendance application! 🎉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rry6C-6ziDw-UerUHnz7g.png" /></figure><h3>Test Health Endpoint</h3><pre>curl $(terraform output -raw alb_url)/login</pre><p>Expected response:</p><pre>{&quot;status&quot;:&quot;healthy&quot;}</pre><h3>OPTIONAL: Adding Custom Domain and SSL</h3><p>If you want to add a custom domain with HTTPS, follow these additional steps.</p><h3>Prerequisites</h3><ul><li>A registered domain name</li><li>Domain managed in Route 53 (or ability to update NS records)</li></ul><h3>Step 16: Configure Route 53 (If Needed)</h3><p>If your domain isn’t already in Route 53:</p><pre># Create hosted zone<br>aws route53 create-hosted-zone \<br>  --name yourdomain.com \<br>  --caller-reference $(date +%s) \<br>  --region eu-west-2</pre><p>Note the name servers and update them at your domain registrar.</p><h3>Step 17: Update Terraform Variables</h3><p>Edit terraform/terraform.tfvars:</p><pre># Uncomment and configure these:<br>domain_name           = &quot;student-portal.yourdomain.com&quot;<br>route53_zone_name     = &quot;yourdomain.com&quot;<br>create_route53_record = true</pre><h3>Step 18: Apply Changes</h3><pre>cd terraform<br>terraform plan<br>terraform apply</pre><p>This will:</p><ol><li>✅ Request an SSL certificate from AWS Certificate Manager</li><li>✅ Automatically validate it via DNS (takes 5–10 minutes)</li><li>✅ Add HTTPS listener (port 443) to ALB</li><li>✅ Configure HTTP → HTTPS redirect (port 80 → 443)</li><li>✅ Create Route 53 A record pointing to ALB</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAOTIJYamLGNAnLB5TFSDw.png" /></figure><h3>Step 19: Verify SSL Configuration</h3><pre># Get your secure URL<br>terraform output application_url</pre><pre># Should show: <a href="https://student-portal.yourdomain.com/">https://student-portal.yourdomain.com</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCxOQ8p2VBdMK8hK-FjZqg.png" /></figure><h3>Test HTTP Redirect</h3><pre># HTTP should redirect to HTTPS<br>curl -I http://yourdomain.com</pre><pre># Expected response:<br># HTTP/1.1 301 Moved Permanently<br># Location: <a href="https://student-portal.yourdomain.com/">https://yourdomain.com/</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QObTUq3jsxwxa2vm-CrwXw.png" /></figure><h3>Test HTTPS</h3><pre>curl https://yourdomain.com/login</pre><h3>Verify Certificate</h3><pre># Check certificate details<br>openssl s_client -connect yourdomain.com:443 -servername yourdomain.com | openssl x509 -noout -dates</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSl3Aen-CWR636hEd-eqgQ.png" /></figure><h3>Troubleshooting</h3><h3>Issue 1: Tasks Not Starting</h3><p><strong>Check stopped tasks:</strong></p><pre>aws ecs list-tasks \<br>  --cluster attendance-app-dev-cluster \<br>  --desired-status STOPPED \<br>  --region eu-west-2<br><br># Describe a stopped task<br>aws ecs describe-tasks \<br>  --cluster attendance-app-dev-cluster \<br>  --tasks &lt;TASK_ARN&gt; \<br>  --region eu-west-2 \<br>  --query &#39;tasks[0].stoppedReason&#39;</pre><p><strong>Common causes:</strong></p><ul><li>Image not found in ECR → Push your Docker image</li><li>Health check failing → Verify /health endpoint works</li><li>Insufficient memory → Increase task resources</li></ul><h3>Issue 2: Target Group Unhealthy</h3><p><strong>Check target health:</strong></p><pre>TG_ARN=$(aws elbv2 describe-target-groups \<br>  --names attendance-app-dev-tg \<br>  --query &#39;TargetGroups[0].TargetGroupArn&#39; \<br>  --output text \<br>  --region eu-west-2)<br><br>aws elbv2 describe-target-health \<br>  --target-group-arn $TG_ARN \<br>  --region eu-west-2</pre><p><strong>Common causes:</strong></p><ul><li>Security group blocking traffic</li><li>Wrong health check path</li><li>Container not listening on port 8000</li></ul><h3>Issue 3: Certificate Validation Stuck</h3><p><strong>If certificate validation takes longer than 10 minutes:</strong></p><ol><li>Check DNS propagation:</li></ol><pre>dig yourdomain.com</pre><ol><li>Verify NS records at registrar match Route 53 nameservers</li><li>Check validation records:</li></ol><pre>aws acm describe-certificate \<br>  --certificate-arn &lt;CERT_ARN&gt; \<br>  --region eu-west-2</pre><h3>Issue 4: Cannot Connect to Database</h3><p><strong>Test from ECS task:</strong></p><pre># Get a task ID<br>TASK_ID=$(aws ecs list-tasks \<br>  --cluster attendance-app-dev-cluster \<br>  --service-name attendance-app-dev-service \<br>  --query &#39;taskArns[0]&#39; \<br>  --output text \<br>  --region eu-west-2)<br><br># Execute command in container<br>aws ecs execute-command \<br>  --cluster attendance-app-dev-cluster \<br>  --task $TASK_ID \<br>  --container attendance-app-container \<br>  --command &quot;/bin/sh&quot; \<br>  --interactive \<br>  --region eu-west-2</pre><p>Then test inside container:</p><pre># Test database connection<br>nc -zv &lt;RDS_ENDPOINT&gt; 5432</pre><h3>Cleanup</h3><p>When you’re done testing, clean up to avoid charges:</p><pre>cd terraform<br># Destroy all resources<br>terraform destroy --auto-approve</pre><p><strong>Note:</strong> This will delete everything including the database. Make sure to backup any important data first!</p><h3>What We’ve Accomplished</h3><p>🎉 <strong>Congratulations!</strong> You’ve successfully deployed a production-ready 3-tier application on AWS using Infrastructure as Code!</p><h3>Infrastructure Created:</h3><p>✅ <strong>Networking:</strong> VPC with public/private subnets across 2 AZs<br> ✅ <strong>Compute:</strong> ECS Fargate cluster with 2 running tasks<br> ✅ <strong>Storage:</strong> RDS PostgreSQL database with automated backups<br> ✅ <strong>Load Balancing:</strong> Application Load Balancer with health checks<br> ✅ <strong>Security:</strong> Defence-in-depth with multiple security groups<br> ✅ <strong>Monitoring:</strong> CloudWatch Logs and Container Insights<br> ✅ <strong>Container Registry:</strong> ECR with image scanning<br> ✅ <strong>Optional:</strong> Custom domain with SSL/TLS certificate</p><h3>Key Benefits:</h3><ul><li><strong>Highly Available</strong> — Multi-AZ deployment</li><li><strong>Scalable</strong> — Can easily adjust task count</li><li><strong>Secure</strong> — Private subnets, encrypted database</li><li><strong>Repeatable</strong> — Infrastructure as Code</li><li><strong>Production-Ready</strong> — Health checks, logging, SSL</li></ul><h3>Next Steps</h3><h3>Part 2: CI/CD with GitHub Actions (Coming Next)</h3><p>In the next article, we’ll automate the entire deployment pipeline:</p><ul><li>✅ Automatic Docker builds on git push</li><li>✅ Automated testing before deployment</li><li>✅ Security scanning for vulnerabilities</li><li>✅ Blue-green deployments</li><li>✅ Rollback capabilities</li><li>✅ Environment-based deployments (dev, staging, prod)</li></ul><h3>Share Your Success!</h3><p>If you successfully deployed your application using this guide:</p><ul><li>Share a screenshot on LinkedIn</li><li>Tag me in your post</li><li>Use hashtag #TerraformECS</li></ul><p>I’d love to see what you’ve built! 🚀</p><p><strong>Connect with me</strong> on <a href="https://www.linkedin.com/in/tobi-akinlade-02b035150/">LinkedIn</a> for more AWS, DevOps, and cloud architecture content. Stay tuned for Part 2 where we’ll add CI/CD automation with GitHub Actions!</p><p>#AWS #Terraform #ECS #DevOps #CloudComputing #InfrastructureAsCode</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ce9e69d6436" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>