Building Astronomy Shop Before Our First Platform Engineering Roles
How two early-career engineers turned self-directed learning into a hands-on cloud-native AWS platform engineering project
There is a big difference between learning Kubernetes from a course and being responsible for a Kubernetes system that keeps breaking.
We wanted to experience that difference ourselves.
So we built Astronomy Shop, a cloud-native commerce platform on AWS, as a two-person engineering project.
We did not start with the goal of claiming that we had built a production platform.
We started with a simpler question:
What would we actually need to build, troubleshoot, secure, monitor, and recover if we wanted to work as platform engineers?
That question shaped almost everything we did.
Getting an application running on EKS was only the beginning. We kept going into GitOps, progressive delivery, autoscaling, security controls, observability, backup and restore, and controlled failure testing.
More importantly, we started documenting not only what worked, but also what failed, what we fixed, and what we still could not prove.
This is the first article in our Astronomy Shop engineering series.
Why We Built It
Learning DevOps tools individually can give you a long list of technologies:
AWS. Terraform. Docker. Kubernetes. GitHub Actions. Helm. Argo CD. Prometheus.
The harder part is understanding how those pieces behave when they have to work together.
That became the purpose of Astronomy Shop.
We wanted to answer practical questions:
- What happens when infrastructure dependencies are created in the wrong order?
- What happens when a Kubernetes controller is missing?
- How does a code change travel from source control into ECR and eventually into EKS?
- What happens when HPA needs more replicas than the current cluster capacity can provide?
- How do we know whether our monitoring actually works?
- Can we restore a Kubernetes volume?
- Can we restore the database?
- What happens when a security control blocks something we expected to work?
Those questions became more useful to us than simply completing another tutorial.
What Astronomy Shop Is
Astronomy Shop is based on the OpenTelemetry Astronomy Shop application.
We used a selected set of its microservices and built our own cloud-native platform around them.
Our Staging environment currently contains 11 workloads:
- frontend-proxy
- frontend
- product-catalog
- cart
- checkout
- payment
- currency
- recommendation
- image-provider
- shipping
The application runs on Amazon EKS.
At a high level, a request travels through:
Browser → Route 53 → CloudFront → AWS WAF → Application Load Balancer → AWS Load Balancer Controller → frontend-proxy → frontend → Internal Microservices → RDS PostgreSQL / Valkey / AWS Services
That describes the application path.
The more interesting part was everything we had to build around it.
The Platform We Built
We separated the project into four repositories.
Infrastructure
astronomy-shop-infrastructure
This repository contains the Terraform used for the AWS foundation and platform components.
That includes networking, EKS, IAM and OIDC, Karpenter, data services, edge components, security-related infrastructure, and observability infrastructure.
Application
astronomy-shop-app
This repository contains the application code, Docker builds, CI workflows, testing, scanning, SBOM generation, and image delivery.
GitOps
astronomy-shop-gitops
This repository contains the desired Kubernetes state.
We use Helm, Argo CD ApplicationSets and AppProjects, environment-specific values, and immutable image references.
Evidence
astronomy-shop-evidence
This repository contains screenshots, terminal outputs, drills, runbooks, and validation material.
We created this repository for a simple reason:
We did not want the project to become a collection of claims.
If we said something worked, we wanted to know what evidence supported it.
That mindset gradually became one of the most important parts of the project.
From a Code Change to a Kubernetes Workload
One of the things we wanted to understand was the complete delivery path.
A code change should not simply become a running container because someone manually executed kubectl apply.
Our intended flow became:
Developer Change → GitHub → GitHub Actions → Build Container Image → Security Scanning → SBOM → Cosign Attestation → Amazon ECR → GitOps Image Digest → Argo CD → Helm → Argo Rollouts → EKS Workloads
This forced us to think carefully about the boundary between CI and CD.
GitHub Actions handles the build and image delivery side.
GitOps contains the desired deployment state.
Argo CD continuously reconciles that desired state.
Helm provides reusable deployment structure.
Argo Rollouts manages rollout behaviour.
Kubernetes runs the workloads.
The important lesson was not memorizing the names of the tools.
It was understanding what each component should own.
Learning Through Problems
The project became much more interesting when things stopped working.
We ran into problems involving Terraform dependencies, EKS access, Karpenter permissions and capacity, missing Kubernetes controllers, Helm rendering, PVC provisioning, application secrets, Valkey connectivity, and container hardening.
One example involved the frontend-proxy container.
We had configured the container with a read-only filesystem, but Envoy still needed to write temporary configuration data.
The solution was not to remove the security control.
We provided the required temporary filesystem through an emptyDir mount and adjusted the container configuration to work with a numeric user.
That was a useful lesson.
A security control should not automatically be treated as the problem simply because the application stops working after the control is enabled.
Sometimes the application needs to adapt to the control.
Our general troubleshooting loop became:
Observed Symptom → Investigation → Root Cause → Controlled Change → Validation → Documentation
We repeated that process throughout the project.
And every time we did, the tools became less important than the reasoning around them.
Security Was Part of the Platform
We did not want security to become a final checklist added after everything else.
The platform includes multiple security layers:
- IAM and workload identity
- AWS Secrets Manager
- KMS
- External Secrets
- Trivy scanning
- CodeQL-related checks
- SBOM generation
- Cosign-related image attestation
- Kyverno
- Falco
- Pod Security Standards
- Kubernetes NetworkPolicies
But we also learned to distinguish between having a control and proving that the control works as intended.
For example, a NetworkPolicy existing in the cluster proves that the policy is deployed.
It does not automatically prove that every intended allow and deny rule has been tested.
Likewise, having Kyverno configured in Enforce mode does not mean every possible policy scenario has been validated.
We started recording those distinctions explicitly.
That made our documentation more honest and, in our view, more useful.
Observability: Installing Components Wasn’t Enough
We deployed Prometheus, Grafana, Alertmanager, Loki, Promtail, OpenTelemetry Collector, Tempo, kube-state-metrics, and node-exporter.
But installing observability tools is much easier than proving that they provide useful information.
We tested several paths.
Prometheus successfully scraped platform components.
Loki returned Staging logs.
A deterministic PrometheusRule generated an alert that travelled through Alertmanager and SNS and reached the mailbox.
CloudWatch also provided cluster, node, and pod-level visibility.
But we found gaps too.
For example, the frontend ServiceMonitor was present, but its /metrics endpoint returned 404.
Get Prasanna M’s stories in your inbox
Join Medium for free to get updates from this writer.
We also found that an expected OpenTelemetry exporter metric query returned no data.
We documented those findings rather than treating the existence of Grafana dashboards as proof that application observability was complete.
That changed how we think about monitoring.
A dashboard can look impressive and still tell you very little when an application actually fails.
Testing HPA and Karpenter
We also wanted to see what Kubernetes autoscaling actually looked like under pressure.
For one controlled Staging experiment, we used Locust with 128 users and a spawn rate of 10.
The frontend started at three replicas.
As CPU pressure increased, HPA moved the workload through:
3 → 4 → 7 → 9 replicas
After the load stopped, the replicas came back down:
9 → 7 → 4 → 3
During the experiment, frontend CPU usage reached roughly 105%, 120%, 146%, and 126% against the configured 65% CPU target.
Karpenter also provisioned capacity as the cluster needed it.
This gave us useful evidence that the autoscaling controllers were reacting to resource pressure.
But there was an important limitation.
The Locust run had approximately 71% request failures.
So we cannot use this experiment to claim that the application successfully handled 128 concurrent users.
What we proved was autoscaling behaviour, not application performance or availability at that load.
That distinction matters.
It is easy to look at a graph showing replicas increasing and conclude that the system handled the workload successfully.
The evidence did not support that stronger claim.
Thinking About Capacity
The autoscaling experiment also made us think about capacity more carefully.
Our frontend Pods used equal resource requests of 250m CPU and 256Mi memory.
At nine replicas, that represents:
9 × 250m CPU = 2.25 vCPU
and
9 × 256Mi = 2.25 GiB
At three replicas:
3 × 250m CPU = 0.75 vCPU
and
3 × 256Mi = 0.75 GiB
So reducing the frontend from nine replicas to three reduces the requested frontend workload capacity by approximately 66.7%.
But that does not mean AWS infrastructure cost fell by 66.7%.
Actual cost depends on node packing, other workloads, Karpenter behaviour, instance types, baseline capacity, and other factors.
We wanted to make that distinction explicit rather than turn a Kubernetes number into a cloud-cost claim.
Recovery Testing
We also treated recovery as something that needed to be exercised rather than assumed.
We completed a Velero backup and restore test.
The restored PVC became bound, and a marker checksum from the restored volume matched the source marker checksum.
We also completed an AWS Backup RDS restore and queried the restored PostgreSQL instance for basic comparison.
These tests gave us evidence that the recovery mechanisms worked for the scenarios we exercised.
They do not give us a production RTO or RPO.
We have not claimed that they do.
That distinction became a recurring theme throughout the project:
A successful test proves the scenario we tested. It does not automatically prove every stronger scenario around it.
What the Project Taught Us
The biggest change was in how we think about engineering work.
At the beginning, it was easy to think in terms of tools:
Learn Terraform.
Learn Kubernetes.
Learn Argo CD.
Learn Prometheus.
After working through the project, our questions became different:
What does this component own?
What happens if it fails?
How do we detect that failure?
What evidence tells us the fix actually worked?
What have we not tested yet?
Those questions feel much closer to the work we want to do professionally.
We also learned that building a platform is not the same as making everything work once.
A configuration can be correct and still fail because a dependency is missing.
A controller can be installed and still be configured incorrectly.
A security control can expose an application assumption.
A dashboard can exist without useful application telemetry.
And an autoscaler can scale correctly while the application is still returning errors.
Those situations gave us more useful learning than simply following a successful tutorial.
Why We Care About Evidence
As the project grew, we started classifying our conclusions.
Implemented means the source or configuration proves the feature exists.
Observed means we captured runtime evidence showing the state.
Exercised means we deliberately performed a test or drill and observed the result.
Unproven means we do not currently have enough evidence to make the stronger claim.
This classification helped us identify unfinished work without treating unfinished work as failure.
For example, Karpenter consolidation has been changed in source configuration from WhenEmpty to WhenEmptyOrUnderutilized with a 10-minute consolidation delay.
We have validated that source change locally.
We have not yet claimed live node removal or actual cost savings from the change until it is applied and naturally observed.
The same principle applies to NetworkPolicy enforcement, complete application tracing, frontend metrics, rollback testing, and performance.
For us, knowing what we cannot yet prove is part of engineering.
The Bigger Goal
Astronomy Shop is not an attempt to simulate years of production experience.
We know the difference.
It is a way for us to get closer to the type of engineering problems we expect to encounter on a real platform team.
The project currently covers six broad areas:
- Cloud and infrastructure foundation
- CI/CD and GitOps delivery
- Kubernetes reliability, scalability, and traffic management
- Security and software supply-chain controls
- Observability and operational visibility
- Backup, recovery, disaster recovery, and cost-aware operations
Each area gave us something different to investigate.
Terraform taught us infrastructure dependency management.
Kubernetes taught us that controllers and workloads have to agree about ownership.
GitOps taught us to separate desired state from runtime state.
Security taught us that controls can expose assumptions in application images and configurations.
Observability taught us to verify telemetry instead of assuming it exists.
Recovery testing taught us that a backup is not the same thing as a successful restore.
And the failures gave us some of the most useful learning of all.
What Comes Next
This is the first article in our Astronomy Shop engineering series.
The next articles will go deeper into the individual engineering areas rather than trying to explain the entire platform at once.
We will cover the AWS foundation with Terraform, the path from source code to a digest-pinned Kubernetes deployment, reusable delivery with Helm and Argo CD, HPA and Karpenter behaviour, security controls, observability, recovery, and failure investigation.
There are also things we still need to prove.
We want to capture genuine Karpenter consolidation behaviour, exercise NetworkPolicy enforcement directly, improve the remaining observability gaps, investigate the high-error load-test result, strengthen CI coverage for the remaining services, and demonstrate safe rollout recovery.
That is part of the project too.
We are not trying to present a finished platform.
We are documenting the engineering process that got us here.
About Us
We are Prasanna M and Narendhiran B, early-career engineers building toward roles in Platform Engineering, DevOps, Cloud Engineering, Kubernetes, and Site Reliability Engineering.
Prasanna M — LinkedIn
Narendhiran B — LinkedIn
Astronomy Shop — GitHub
Astronomy Shop has given us a practical environment in which to build, troubleshoot, test, document, and question our own assumptions.
We are not presenting the project as a substitute for production experience.
Instead, we see it as evidence of how we approach unfamiliar engineering problems: by building, investigating failures, validating changes, documenting evidence, and being explicit about what remains unproven.
We are open to opportunities and conversations with engineering teams that value reliable execution, continuous learning, and evidence-led engineering.
About This Series
Part 1 — Building Astronomy Shop Before Our First Platform Engineering Roles
Part 2 — Terraforming the AWS Foundation for a Cloud-Native Commerce Platform
Part 3 — From Source Code to a Digest-Pinned Kubernetes Deployment
Part 4 — Designing Reusable Kubernetes Delivery Controls with Helm, Argo CD, and Rollouts
Part 5 — Proving Autoscaling in Staging with HPA and Karpenter
Part 6 — Karpenter Consolidation: Diagnosing Capacity Efficiency Safely
Part 7 — Building Defence in Depth: CI Security, Runtime Detection, and Network Policies
Part 8 — Observability Without Dashboard Theatre: Metrics, Logs, Alerts, and CloudWatch
Part 9 — The Failures That Made Us Better Platform Engineers
Part 10 — Evidence-Led Platform Engineering: What Astronomy Shop Demonstrates
