How a hard API limit of 200,000 requests per day in Sonatype Nexus OSS forced a repository strategy rethink — and how moving language package ecosystems to a managed AWS service replaced a $15,000-per-year Nexus Pro upgrade with a $60–$90 monthly bill

Every platform team running a self-hosted artifact registry eventually hits the same wall. The registry is a silent dependency for every build, deployment, and base-image refresh. The moment it slows down, the entire delivery pipeline slows with it. In a regulated fintech environment, where supply-chain control and build reproducibility are not optional, the wall arrives sooner.

For us, the wall took the form of HTTP 402 Payment Required.

This article describes how a single rate limit on our self-hosted Sonatype Nexus OSS instance forced a full re-evaluation of the artifact registry stack. It explains why we chose AWS CodeArtifact over the obvious commercial alternatives, and how a $15,000-per-year licensing decision was replaced with a roughly $60–$90 monthly managed-service bill — without sacrificing developer velocity, auditability, or support for the formats AWS does not yet cover.

The Challenge: When the Registry Becomes the Bottleneck

For years, Nexus Repository OSS served as the single artifact registry for the platform. It hosted every format the engineering organisation depended on: npm and NuGet for application packages, PyPI for Python tooling, Docker images for application and base images, APT and YUM for operating-system packages, Helm charts for Kubernetes deployments, and raw GitHub proxies for upstream sources. At peak it served roughly 30 repositories across hosted, proxy, and group types.

After upgrading to a recent Nexus OSS version, we ran into the hard API limit of 200,000 requests per day introduced in newer OSS releases. For a platform serving nearly 350+ active projects with continuous integration, container rebuilds, scheduled dependency refreshes, and parallel pipeline runners, that ceiling stopped being theoretical:

• CI pipelines started returning HTTP 402 Payment Required mid-build, with no graceful degradation
• Proxy pulls from upstream registries (npmjs.org, NuGet.org, PyPI) stopped working
• npm install, dotnet restore, and pip install failed deterministically once the daily quota was exhausted
• Pushes of internal packages to hosted repositories were blocked alongside the proxy traffic
• Automatic retries compounded the problem by consuming additional quota
• Recovery required waiting for the daily counter reset — there was no operational workaround once the limit was hit.

Additional caching layers in front of Nexus could theoretically have absorbed the excess traffic. The operational reality, however, was another stateful service, another point of failure, another set of credentials to rotate, and another item added to the audit trail. In a regulated environment, every component added to the supply chain is also a component that has to be defended during an audit.

The real question was no longer how to work around the limit, but whether we should be self-hosting this at all.

Alternatives Evaluated

Four options were on the table.

In practice, the real choice was between Nexus Pro and AWS CodeArtifact. JFrog and Pulp were eliminated early: JFrog due to total cost of ownership, and Pulp due to insufficient operational maturity for a fintech context. The decision was captured as an internal Architecture Decision Record before any migration work began.

The decisive factor was not feature parity. It was the realisation that the artifact registry is infrastructure, not a product we wanted to operate. Every hour spent patching Nexus, planning version upgrades, monitoring disk usage, and handling rate-limit incidents was an hour not spent on the EKS platform itself. A managed service shifts that work to AWS.

The trade-off was explicit and deliberate: CodeArtifact covers the main package ecosystems we depend on through poxy repositories — npm, NuGet, and PyPI — alongside Maven, Cargo, RubyGems, and several others we did not need. It does not support Docker images, APT, YUM, or Helm. We accepted that trade-off by migrating only the formats for which CodeArtifact provides proxy support and keeping Nexus running as a scaled-down fallback for everything else.

Architecture Overview: A Two-Tier Registry Model

The post-migration design is a deliberate two-tier model where each tier does what it does best.

At a high level:

• GitLab CI runners on EC2 assume IAM roles to authenticate against AWS CodeArtifact for npm, NuGet, and PyPI traffic
• CodeArtifact and its S3 back-end are reached over VPC Interface and Gateway Endpoints, keeping artifact traffic within the AWS network with no internet egress
• The downsized Nexus instance continues to serve Docker images, APT, YUM, Helm, and raw GitHub proxies over the existing private VPC route
• CodeArtifact domains, repositories, and access policies are managed through Terraform alongside the rest of the platform infrastructure

The split between the two tiers is drawn along a clear line:

• AWS CodeArtifact serves the high-volume language-package formats that were generating the bulk of the traffic that triggered the 402: npm, NuGet, and PyPI. These are the formats every microservice build touches on every pipeline run, several times per pipeline.
• Nexus (downsized) continues to serve the formats AWS does not yet manage natively: Docker images and proxies (Docker Hub, GitHub Container Registry, Quay, public.ecr.aws, Kubernetes registry, Microsoft images), APT (Debian Bookworm, Trixie, Ubuntu Noble), YUM (Fedora), Helm charts, and raw GitHub proxies.

Moving the noisy formats to CodeArtifact removed the daily-quota pressure from Nexus. The Nexus instance type could then be reduced, the disk pressure dropped, and the operational risk of a single self-hosted registry was contained to the formats that did not have a managed alternative.

The design philosophy is straightforward: delegate to managed services where they exist, self-host only what cannot be delegated, and size each tier to the actual traffic it carries.

Migrating Under Pressure: Biggest Consumer First

The migration was not executed as a stop-the-world cutover. It could not be — production builds were already failing intermittently.

Instead, the migration was sequenced by traffic share. The single largest CI consumer of Nexus quota was migrated to CodeArtifact in the first week after the ADR was accepted. That one project, once cut over, removed enough load from Nexus to bring the daily counter back below the 200,000 ceiling. The pressure dropped, builds stopped failing, and the remaining 340+ projects migrated in the following weeks at a normal cadence.

This sequencing turned out to be the most important operational decision in the entire project. A traditional “migrate everything, then switch” plan would have left the platform broken for weeks. Migrating the heaviest consumer first turned a production incident into a controlled rollout.

For each project, the migration changed three things:

• The CI pipeline gained an aws codeartifact login step, executed with the runner’s IAM role
• Project configuration files (.npmrc, pip.conf, NuGet.config) were rewritten to point at the CodeArtifact repository URL
• The base image used by the runner was updated to include the AWS CLI, so the login step would succeed without per-project installation

For the developer, the change was visible only as a different repository URL in the project configuration. The authentication, refresh, and credential rotation were all handled by IAM and the CodeArtifact login command. There were no long-lived tokens to manage and no service accounts to rotate.

Benefits included:

• No further HTTP 402 failures on the migrated formats
• Authentication via IAM roles attached to GitLab runners on EC2, with no long-lived tokens to manage
• Credential rotation handled by AWS, not by the platform team
• Per-project access scoped by IAM policy, auditable through CloudTrail
• Migration could proceed project-by-project without a coordinated downtime window

Authentication and Networking: IAM as the Control Plane

The most significant operational change was not the registry itself. It was the authentication model.

Under Nexus, every CI runner used long-lived credentials stored in pipeline variables and rotated them manually. Under CodeArtifact, GitLab runners assume IAM roles and exchange them for short-lived authentication tokens through the aws codeartifact login command, which writes the token into the relevant package-manager configuration for the duration of the build.

The networking model is equally direct. CodeArtifact and its S3 back-end are reached over a VPC interface endpoint (PrivateLink) for CodeArtifact, plus an S3 gateway endpoint for the backing store, keeping all artifact traffic on the AWS backbone. There is no internet egress for package pulls, which both reduces NAT costs and tightens the network audit boundary.

This matters for three reasons which compound in a regulated environment:

• No long-lived tokens. A compromised runner credential is bounded by the IAM session, not by manual rotation cadence.
• Audit trail by default. Every GetAuthorizationToken, every package read, and every package published appears in CloudTrail, tagged with the assuming role. Audit responses go from “we’ll dig through Nexus logs” to “here is the CloudTrail query.”
• Permissions as code. Repository access is granted through IAM policies managed in Terraform, alongside the rest of the platform’s infrastructure. Adding a new project to the registry is a pull request, not a click in a Nexus admin console.

Cost: From License Negotiation to a Utility Bill

A common assumption is that managed services are more expensive than self-hosting. For artifact registries at our scale, the opposite is true.

The Nexus Pro license alone would have cost approximately $15,000 per year. That number does not include the EC2 instance, EBS storage, backups, patching effort, version upgrade work, monitoring, or the on-call burden of disk-pressure incidents, all of which we were already paying for under Nexus OSS.

CodeArtifact, after absorbing the npm, NuGet, and PyPI workload for all ~350 projects, costs us roughly $60–$90 per month, billed on storage and request volume. That is about $720–$1,080 per year, against the $15,000 license figure — and that comparison still understates the win because it ignores the EC2 hours and engineering time that left the books at the same time.

Three structural reasons underlie the cost change:

• Pay-per-request pricing. Idle repositories cost nothing beyond storage. There is no fixed minimum.
• Native lifecycle policies. Old versions are pruned automatically, keeping storage costs flat.
• No infrastructure to right-size. The EC2 instance that Nexus ran on could be downsized once the high-traffic formats left, recovering further compute spend.

Cost and security are not opposites in this design. The pricing model that makes CodeArtifact cheap is the same model that makes it auditable. Every request is metered; every request is logged.

Lessons Learned

Several lessons came out of this migration that are worth capturing for any team considering the same move.

• A rate limit is an architectural signal, not just an operational problem. When a self-hosted service introduces a hard ceiling that breaks production, the right response is rarely to engineer around the ceiling. It is to ask whether self-hosting was the right call in the first place. Caching layers and quota dashboards would have bought time; they would not have removed the underlying ownership cost.
• Migrate the loudest consumer first. In a 350-project migration triggered by a live incident, sequencing matters more than completeness. Cutting over the single biggest traffic source in week one returned headroom to the old system and bought the time needed to migrate the rest at a normal pace. A stop-the-world cutover would have extended the outage.
• Accept partial managed-service coverage. CodeArtifact does not support every artifact format. Trying to find a single managed service that covers Docker, APT, YUM, Helm, and language packages would have meant either staying on Nexus or moving to a costlier commercial platform. Splitting the stack — managed for what’s covered, self-hosted for the rest — is a more honest architecture than pretending a single service can do everything.
• IAM replaces a whole category of credential operations. The largest operational simplification was not the registry itself. It was the disappearance of long-lived tokens and the rotation policies that surround them. In a regulated environment, every long-lived credential is also an audit obligation. Replacing them with short-lived IAM-derived tokens removes that obligation.
• Managed services move work, not eliminate it. CodeArtifact removed the EC2 patching, the disk monitoring, and the upgrade planning. It did not remove the need to think about access control, repository design, or lifecycle policies. The work is different, not absent.

Recommendations for Teams Considering a Similar Move

Teams evaluating whether to leave a self-hosted artifact registry behind can reduce risk by adopting a few principles early.

Start by inventorying traffic by format, not by repository count. The format generating the most requests is almost always a language-package proxy, and almost always the one a managed service like CodeArtifact will cover. The migration ROI is concentrated there.

Treat the artifact registry as part of the audit boundary. Every credential, every endpoint, and every log path is part of the supply-chain story for compliance reviewers. Managed services with native IAM and CloudTrail integration shorten that story.

Adopt Infrastructure as Code from the first repository. Manage CodeArtifact domains, repositories, and access policies through Terraform from the start. Click-ops registries become un-auditable registries within a year.

Sequence the migration by traffic, not by team boundary. A migration triggered by a production incident is not a tidy project; it is an incident response. Move the biggest consumer first to relieve the pressure on the legacy system, then let the rest follow.

Finally, keep the legacy system alive as a downsized fallback for formats the managed service does not cover. Pretending the old system is gone is a marketing claim. Keeping it operational, smaller, and clearly scoped is an engineering decision.

What’s Next

The next logical step is to extend the managed-service model to container images. Amazon Elastic Container Registry (ECR) is the natural target for the Docker workload still served by Nexus, with ECR pull-through cache replacing the upstream proxies for Docker Hub, GitHub Container Registry, Quay, and the public AWS registry. Helm chart distribution moves to ECR’s OCI artifact support in the same step.

Once Docker leaves Nexus, the remaining Nexus footprint is narrow: APT, YUM, and raw GitHub proxies. At that point, the question of whether to retire Nexus entirely — and replace the remaining formats with a thinner self-hosted alternative or a different managed pattern — becomes worth asking.

Conclusion

The mandate that started this work was straightforward: keep CI building, keep developers shipping, and do it without writing a $15,000 cheque to retain a self-hosted platform we were already maintaining. The result was not a single replacement product but a deliberate split — a managed AWS service for the formats it covers, a downsized self-hosted instance for the rest, and a clear roadmap for narrowing the self-hosted footprint over time.

The key takeaway here is that the choice between self-hosted and managed is rarely binary and rarely permanent. It is a question to be re-asked every time the underlying service changes — whether the change is a new rate limit, a new pricing tier, or a new managed alternative. In our case, a single HTTP status code was enough to reopen the question. The answer, this time, was AWS CodeArtifact.

Designing a production-grade, zero-internet-access environment with Amazon EKS and AWS managed services

Financial institutions face a structural tension: they are expected to deliver modern, cloud-native applications at speed while operating under some of the industry’s most stringent security and compliance requirements.

For many fintech and banking platforms, this tension resolves into a clear mandate: production workloads must not have direct internet access. Eliminating internet connectivity significantly reduces the attack surface and helps meet regulatory expectations — but it also raises difficult questions.

• How do you support modern CI/CD pipelines?
• How do you enable fast release cycles?
• How do developers remain productive without direct access to infrastructure?

This article outlines how we designed and deployed a fully internet-isolated, production-grade fintech platform on AWS, running multiple microservices on Amazon EKS. The result is an environment that meets regulatory and security requirements without compromising operational efficiency or developer velocity.

The Challenge: Security Without Slowing Delivery

Our client operates mission-critical banking applications and requires a platform with strict constraints:

• No direct internet access for workloads
• No NAT Gateway or Internet Gateway in production VPCs
• Full isolation from corporate and banking networks
• Complete auditability of all changes and network flows
• Support for modern DevOps and GitOps workflows

At the same time, the platform needed to support external customer-facing services and integrate with internal banking systems, while maintaining tight perimeter controls. Meeting these requirements demanded more than tightening security controls — it required re-thinking network, access, and operational design from first principles.

Architecture Overview: A Private, Controlled Core

We designed the platform around a private, internet-isolated core. All ingress and egress traffic flows through explicit, controlled entry and exit points, with no implicit connectivity.

At a high level:

• External traffic enters through AWS WAF and an Application Load Balancer (ALB), protected by AWS Shield and TLS certificates managed by AWS Certificate Manager.
• Traffic is forwarded to a Network Load Balancer (NLB) provisioned by Istio, which handles internal routing into the service mesh.
• Internal services communicate using Amazon Route 53 Private Hosted Zones, maintaining clear separation between public and private resolution.
• All outbound traffic exits through AWS Transit Gateway and dedicated hardware firewalls, providing a single inspection and logging point.

There are no NAT Gateways, no Internet Gateways, and no public EKS endpoints. This design ensures every network path is intentional, observable, and auditable.

Enforcing Isolation at the Pod Network Layer

In regulated environments, isolation cannot rely solely on policy. It must be enforced at the infrastructure level.

One of our most critical requirements was to ensure that pod IP addresses could never be routed into corporate or banking networks — even accidentally.

Using Non-Routable CGNAT Space

We configured the VPC CNI plugin to assign pod IP addresses from a non-routable CGNAT range:

Pod CIDR: 10.0.0.0/16

This range is intentionally not allocated through AWS IPAM and is not routable across the Transit Gateway. As a result:

• Pod traffic cannot leak into internal networks
• Internal WAN routes never overlap with pod addresses
• Routing boundaries create an additional isolation layer
• The EKS cluster has virtually unlimited pod IP capacity

VPC CNI Custom Networking

To implement this model, we used VPC CNI custom networking with ENIConfig resources per Availability Zone and node group. Each worker node uses an ENIConfig that references dedicated subnets mapped to the 10.0.0.0/.16 range. This configuration allows us to:

• Separate node and pod routing paths
• Maintain deterministic pod egress flow toward the Transit Gateway and firewalls
• Apply strict routing and firewall policies
• Simplify troubleshooting with clean IP separation

ENI Prefix Delegation

For better scalability, we enabled ENI prefix delegation. Each ENI receives a /28 prefix, providing up to 16 pod IPs per prefix. The benefits:

• Far fewer Amazon EC2 API calls
• Faster pod scheduling and accelerated node scale-out
• Higher pod density per node
• Lower operational overhead

Configuration Details

We run the VPC CNI plugin in custom network configuration mode:

AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

AWS_VPC_K8S_CNI_EXTERNALSNAT=false

Setting AWS_VPC_K8S_CNI_EXTERNALSNAT=false ensures the node performs SNAT. This enables fully deterministic egress through the firewall while preserving pod identity visibility for the Security Operations Center (SOC) and audit systems.

Routing Isolation as a Security Control

Because 10.0.0.0/16 is non-routable through Transit Gateway:

• Pod subnets are physically unreachable by internal networks
• No accidental trust relationships can form
• Zero risk of IP overlap with banking systems
• Routing boundaries enforce least privilege by default

Application egress still works normally. Pod traffic is SNATed on the node, and egress leaves the VPC using the node’s primary VPC IP address. This address is routable through the Transit Gateway and inspected at the hardware firewall.

This design combines pod-level isolation with controlled egress and full auditability — a hardened network posture aligned with financial regulatory requirements.

DNS as a Security Control

DNS is often overlooked — but in isolated environments, it becomes a critical control plane.

Instead of using Amazon Route 53 Resolver for inbound and outbound queries, we deployed EC2-based DNS resolvers with static IPs. These resolvers:

• Host authoritative records for platform-specific internal domain
• Forward queries for broader internal namespaces
• Maintain predictable behavior for firewall rules and monitoring

This model gives us complete control over DNS behavior, change management, and inspection, while maintaining the benefits of AWS private networking.

Access Model: Zero Direct Human Access

To maintain auditability and reduce risk, we adopted a zero-direct-access model for both production and test environments.

• Developers do not access Amazon EKS clusters directly
• All changes flow through GitOps pipelines using Argo CD
• Amazon EKS API endpoints are private and inaccessible from the internet

This model simplifies compliance, improves traceability, and removes an entire class of operational risk — while still enabling rapid iteration through automated pipelines.

Security Built Into the Platform

Security controls are layered and automated throughout the environment:

• AWS WAF and AWS Shield protect external entry points
• Service Control Policies (SCPs) restrict unnecessary AWS API usage
• AWS CloudTrail, AWS Config, and GuardDuty provide continuous auditing
• Encryption is enforced using AWS KMS customer-managed keys
• Automated compliance checks and remediation run continuously
• Scheduled security audits are executed via Prowler

The platform successfully passed the AWS Foundational Technical Review (FTR) and a dedicated Amazon EKS security audit, validating the architecture against AWS best practices.

Automation and CI/CD in a Private World

Internet isolation does not eliminate CI/CD — it changes how it’s designed.

All pipelines run on private infrastructure and interact exclusively through VPC endpoints and internal services. Key elements include:

• Golden AMIs built and signed using EC2 Image Builder
• Automated patching and configuration via AWS Systems Manager
• Image scanning and secret detection integrated into CI workflows
• Amazon EKS Pod Identity for secure, token-free IAM access

The result is a delivery pipeline that is fully automated, fully auditable, and entirely private.

FinOps: Making Private Infrastructure Cost-Effective

A common misconception is that private environments are inherently expensive. In practice, cost discipline must be intentional.

We applied FinOps principles from the start:

• Karpenter dynamically right-sizes EKS compute capacity
• AWS Graviton instances reduce compute costs for compatible workloads
• Managed services (ECR, CodeArtifact) replace self-hosted tooling
• Eliminating NAT Gateways significantly reduces egress costs
• AWS Budgets and cost allocation tags provide visibility and control

Security and cost efficiency are not opposites — when designed together, they reinforce each other.

Lessons Learned

Several important themes emerged during the design and operation of this platform, each reinforcing the criticality of early architectural decisions in regulated environments.

• Network design comes first. Operating without Internet Gateways or NAT Gateways fundamentally shapes every downstream decision, from CI/CD pipelines to DNS resolution and service connectivity. Treating the network as an afterthought would have resulted in costly rework. By designing routing, ingress, egress, and VPC endpoints upfront, we were able to build security into the foundation rather than layering it on later.
• GitOps changes culture, not just tooling. Removing direct human access to clusters improves security and auditability, but it also requires a shift in how teams work. Developers must trust automation and pipelines rather than rely on ad hoc fixes. Clear communication, training, and strong feedback loops were essential to making this transition successful and ensuring teams remained productive.
• DNS is foundational infrastructure. In an internet-isolated environment, DNS becomes a critical control plane rather than a background service. Taking ownership of DNS behavior increased our operational responsibilities, particularly around availability and scaling, but it also provided predictability, stronger security boundaries, and better alignment with firewall and monitoring requirements.
• Security is a continuous process. Passing an audit is not the end state. Maintaining a strong security posture requires constant validation, automated controls, and regular testing. Tools like AWS Config, Prowler, and Service Control Policies helped ensure that security expectations were enforced consistently over time, even as the platform evolved.
• Private environments don’t have to be slow or expensive. With the right architectural choices, automation, and cost controls, a fully private platform can still support fast delivery and remain cost-efficient.

Technologies such as Karpenter, AWS Graviton, and managed AWS services played a key role in balancing security, performance, and cost.

Recommendations for Regulated AWS Environments

Teams designing regulated platforms on AWS can reduce risk and complexity by adopting a few foundational practices early in their journey.

Start by designing network and access models before building applications. Decisions around routing, ingress, egress, DNS, and identity have far-reaching implications and are much harder to change once workloads are in production. A clear upfront design creates a stable foundation for both security and scalability.

Treat DNS, routing, and identity as first-class security controls, not just supporting services. In tightly controlled environments, these layers define trust boundaries and determine what can communicate with what. Investing in their design pays dividends in auditability and operational clarity.

Adopting GitOps as the primary operational model significantly reduces human risk while improving traceability. When all changes flow through version-controlled pipelines, audits become simpler, rollbacks become safer, and environments remain consistent.

Automation should be embedded from day one. Compliance checks, security audits, and tagging enforcement are far more effective when they run continuously rather than as periodic manual reviews. Automation helps teams maintain compliance as the platform evolves.

Finally, use AWS managed services wherever possible. Managed services reduce operational overhead, improve reliability, and enable teams to focus on security, compliance, and application delivery rather than on infrastructure maintenance.

Conclusion

Building a fintech platform on AWS requires balancing security, compliance, and developer velocity. By eliminating direct internet access, using AWS managed services, and automating security and cost controls, we created a resilient, auditable, and cost-efficient environment for highly regulated workloads.

If you’re working in a similar space, start with network and access design. Integrate security controls into your development process, not as an afterthought. Use services like VPC endpoints, AWS Shield, Amazon GuardDuty, AWS Config, and Karpenter to simplify operations. Adopt GitOps and automate everything you can.

This approach proves that tight security controls and fast development cycles aren’t mutually exclusive — they just need the right architecture.

Machine learning (ML) teams often face a familiar challenge: balancing compute performance with infrastructure costs. Model training can take hours or even days of high-performance computing, while inference and orchestration workloads typically require far less capacity.

Static Kubernetes clusters often lead to overprovisioned environments that waste money or underprovisioned ones that cause performance bottlenecks.

In this post, we demonstrate how to use Amazon Elastic Kubernetes Service (Amazon EKS) with Karpenter, an open-source Kubernetes cluster autoscaler, to dynamically provision compute nodes on demand. This approach enables organizations to match compute resources to workload requirements in real-time, helping to improve performance and reduce costs for MLOps workloads on AWS.

Why Static Node Groups Drive Up MLOps Costs

Most Kubernetes clusters on AWS use static node groups — predefined Amazon Elastic Compute Cloud (Amazon EC2) instances that run continuously. For ML platforms such as Kubeflow, static nodes create several operational challenges:

• Intermittent high-demand workloads: Training jobs require GPU or high-memory instances only during specific windows.
• Variable inference demand: Inference workloads scale up or down with unpredictable external requests.
• Bursty development pipelines: Data preprocessing and testing often produce short-lived, high-intensity workloads.

Organizations face a tradeoff between paying for idle capacity or risking resource shortages that slow experimentation and deployment.

Case Study: Scaling Forecasting Workloads Efficiently

A retail technology company built a forecasting platform using Kubeflow on Amazon EKS to predict daily sales for more than 10 million item-store-date combinations within a month. The system processes over 10 GB of Parquet files per run and executes up to 300 concurrent forecasting jobs. The computational requirements vary significantly throughout the day:

• Model training: Runs for approximately 4 hours with high CPU and memory utilization.
• Forecast generation: Executes in parallel for about 1 hour with moderate compute demand.
• Idle periods: Require minimal resources for orchestration and monitoring.

With static node groups, the company would need to maintain sufficient capacity to support peak loads — 300 concurrent jobs — 24 hours a day, resulting in substantial idle costs during periods of low activity.

The following architecture illustrates how these components work together to deliver a scalable and cost-efficient MLOps platform on Amazon EKS:

How Karpenter Enables Dynamic and Cost-Efficient Provisioning

Karpenter is an open-source Kubernetes cluster autoscaler that automatically provisions compute nodes based on pending pod requirements. Unlike the standard Kubernetes Cluster Autoscaler, which scales predefined node groups, Karpenter directly launches Amazon EC2 instances that match the exact resource needs of your workloads.

For machine learning workflows, Karpenter provides the following benefits:

When running Kubeflow pipelines on Amazon EKS, each step can specify its own resource requirements. Karpenter provisions and retires nodes accordingly, ensuring that the cluster footprint scales precisely with workload demand.

For the forecasting platform, this means:

• Automatically scaling up to 300 nodes during peak forecasting periods.
• Running high-memory or GPU instances only during training.
• Scaling back to a minimal footprint during idle periods.

Organizing Dynamic Nodes with Clear Naming Conventions

Dynamic node provisioning introduces new operational challenges — particularly in identifying which nodes serve specific workloads. Consequently, without consistent organization, troubleshooting, capacity planning, and cost attribution become more complex.

A simple prefix-based naming convention can address this. By categorizing nodes by capacity (for example, S-, M-, L-, or XL-), teams can quickly:

• Gain visibility: Understand node capabilities at a glance in cluster views.
• Simplify debugging: Trace jobs and performance metrics to specific resource tiers.
• Improve cost tracking: Align node groups with workload categories in AWS Cost Explorer.

In the forecasting platform example, small (S-) nodes handle orchestration and lightweight processing, while larger (L- and XL-) nodes manage data-intensive training and forecasting workloads.

To implement this naming convention in the cluster, you can define a dedicated Karpenter NodePool and NodeClass for each node tier, as shown below:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: s-nodes
spec:
  template:
    metadata:
      labels:
        node.kubernetes.io/type: "s"
    spec:
      nodeClassRef:
        name: s-nodeclass
      requirements:
        - key: "karpenter.k8s.aws/instance-family"
          operator: In
          values: ["m6", "m6a"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
  limits:
    cpu: 100
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: s-nodeclass
spec:
  amiFamily: AL2023
  role: "KarpenterNodeRole"
  subnetSelector:
    karpenter.sh/discovery: "eks-demo"
  securityGroupSelector:
    karpenter.sh/discovery: "eks-demo"

To automatically route workloads to the correct node tier based on their prefix, you can use a Kyverno policy that adds the appropriate node affinity to matching pods:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: apply-node-affinity-based-on-pod-prefix
spec:
  mutateExistingOnPolicyUpdate: true
  rules:
    - name: add-affinity-s
      match:
        resources:
          kinds:
            - Pod
          name: "s-*"
      mutate:
        patchStrategicMerge:
          spec:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                    - matchExpressions:
                        - key: node.kubernetes.io/type
                          operator: In
                          values:
                            - "s"

    - name: add-affinity-m
      match:
        resources:
          kinds:
            - Pod
          name: "m-*"
      mutate:
        patchStrategicMerge:
          spec:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                    - matchExpressions:
                        - key: node.kubernetes.io/type
                          operator: In
                          values:
                            - "m"

    - name: add-affinity-l
      match:
        resources:
          kinds:
            - Pod
          name: "l-*"
      mutate:
        patchStrategicMerge:
          spec:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                    - matchExpressions:
                        - key: node.kubernetes.io/type
                          operator: In
                          values:
                            - "l"

Measuring the Cost and Performance Benefits

Organizations adopting Karpenter with Amazon EKS often observe the following improvements:

• Lower idle costs: Nodes exist only during active workloads.
• Better instance matching: Compute is right-sized to each job, minimizing waste.
• Higher utilization: Resource consumption closely tracks demand.
• Elastic capacity: Large experiments scale up temporarily, then automatically scale down.

In the retail forecasting use case, the company achieved an estimated 30–50% reduction in compute costs by combining Karpenter, Amazon EC2 Spot Instances, and Savings Plans. Dynamic autoscaling eliminated idle capacity costs while maintaining service-level objectives during peak workloads.

Beyond cost optimization, the platform also reduced model deployment cycles — from weeks to hours — enabling data scientists to iterate faster and improve model quality.

Building a Scalable, Cost-Optimized MLOps Platform

Dynamic node provisioning works best as part of an integrated, managed MLOps architecture. The forecasting platform combined several AWS services to achieve scalability, reliability, and cost efficiency:

• Amazon EKS provides the managed Kubernetes control plane, removing cluster management overhead.
• Amazon Aurora serves as the metadata store for Kubeflow, decoupling pipeline state from the cluster and improving fault tolerance.
• Amazon Simple Storage Service (Amazon S3) stores raw data, features, and inference results. Pipelines read and write Parquet files directly from S3, while Karpenter scales compute capacity as needed.
• AWS Identity and Access Management (IAM) and AWS Secrets Manager manage authentication and secret rotation, integrated with on-premises identity systems through AWS Direct Connect.
• Amazon CloudWatch and AWS Budgets offer observability and cost management, providing alerts to prevent unexpected expenses.

This architecture allowed the company to deploy workloads in private subnets, isolate sensitive components, and maintain production-grade reliability — all while minimizing infrastructure overhead.

Best Practices for Configuring Dynamic Node Provisioning

To maximize efficiency and control costs when implementing dynamic node provisioning:

1. Set node termination policies carefully: Configure Karpenter to remove idle nodes promptly but safely.
2. Standardize naming conventions: Use consistent prefixes (S-, M-, L-, XL-) for clarity and cost tracking.
3. Define resource boundaries: Set maximum vCPU and memory limits per node type to avoid runaway costs.
4. Specify pod resource requests: Define CPU, memory, and GPU needs in Kubernetes manifests to ensure optimal instance selection.
5. Monitor continuously: Use Amazon CloudWatch, Prometheus, or Grafana to track utilization and identify opportunities for optimization.
6. Use Spot Instances for fault-tolerant jobs: Combine Amazon EC2 Spot Instances with Karpenter’s instance diversity for additional savings.

Conclusion: Dynamic Provisioning as a Catalyst for Scalable MLOps

By combining Amazon EKS with Karpenter, organizations can dynamically match compute capacity to workload requirements — reducing costs while maintaining the agility that modern ML teams need.

The retail forecasting platform described in this post demonstrates these benefits in production:

• Processing more than 10 million daily predictions
• Running hundreds of concurrent jobs
• Achieving 30–50% compute cost savings while reducing deployment cycles from weeks to hours.

For teams running Kubeflow or similar machine learning platforms on AWS, dynamic node provisioning provides a practical and cost-effective path to scalable and efficient MLOps.

In our previous post, we described how the migration process started; in today’s post, we’d like to share a few recent achievements. We received a higher AWS Partner tier — Advanced and got dozens of AWS Certifications.

This summer, our elKasa project has successfully passed the AWS Foundational Technical Review (FTR) and earned the AWS Qualified Software status. Here is how we progressed with the project after its launch and how it will benefit from the new status.

elKasa: Project Background

We ventured into migration from on-prem to AWS infrastructure to respond to some security challenges, but it was not just data security that we achieved. Cloud computing services by Amazon allowed us new perspectives and a powerful impetus to advancement, eventually leading to a radical transformation in both development and operational mindsets. Our engineers got certified by AWS, which, in turn, enriched the arsenal of approaches and best practices we use for software development.

In late 2023, we launched a new AWS-based service — elKasa. In a nutshell, it replaces hardware Payment Transaction Registers (PTR) with Software Payment Transaction Registers (SPTR) and allows administering them from the cloud. You can find more information about this project in this post.

Using elKasa, we can now issue electronic receipts to our Guests. The switch to SPTR happens seamlessly as, while integrating the solutions, we can not stop the store operations. We pride ourselves in addressing this technical challenge.

As of now, elKasa is used in four entities of the Fozzy Group holding: the Silpo and Fora supermarket chains, THRASH! stores and Fozzy Cash&Carry hypermarkets. The elKasa service is available in all Silpo and Fora supermarkets — 4466 SPTR in 316 Siplo supermarkets and 2408 SPTR in 328 Fora stores, and counting!

elKasa was designed to securely store the data, automate report generation and staff management, take care of the environment and optimize costs, reduce the waiting time at the cash register, and keep access to online order receipts.

How We Received the AWS ‘Qualified Software’ Status

To receive the AWS Qualified Software status, every application needs to go through the AWS Foundational Technical Review.

We evaluated elKasa’s architecture and submitted the documentation on the project to an Amazon architect, who confirmed that the app matches the quality attributes and cloud architecture standards.

Earning the ‘Qualified Software’ status was not our goal in itself. Since quality software is our priority, the elKasa team just wanted to validate whether they had chosen the right approach to development. Now it is safe to say that, elKasa’s architectural decisions and development approaches meet the modern cloud standards.

The Benefits of the AWS Qualified Software Status

The AWS Qualified Software status is valid for three years, so elKasa will have it expire after the 27th of July, 2027. It is worth mentioning that, as a part of the technical review process, the application’s architecture and operations are to be periodically reviewed against the given standards. This enables us to identify gaps and continually improve the solution.

By earning the AWS Qualified Software status, we proved that:

• our application meets the current standards and innovative requirements;
• we are qualified to modernize other platforms and applications following the cloud standards;
• our architects and engineers see maximization of the cloud benefits as their priority;
• we are ready to handle various unpredictable situations along the software development cycle.

What’s Next?

Though the AWS Qualified Software status will not be taken away, we do not plan to stop at the top. Cloud transformation of our applications continues in full swing. Let us now forget that even the most modern systems become obsolete and ineffective over time, so the improvement process is endless.

While changing the development process, we also transformed our approach to operations (i.e., the way we handle projects and workload). For instance, we adopted best SRE practices for SLA contracts in our applications, implemented modern FinOps approaches and conducted dedicated training on the FinOps framework for the project teams.

We have developed Disaster Recovery plans (DRP) for all projects to prevent force majeures and/or mitigate their negative consequences. We conduct a detailed post-mortem analysis of every incident, outlining the ways to address and mitigate them in the future.

Our cloud transformation goes on, despite the path being thorny. We plan to integrate elKasa for more stores and business entities. Our software engineers continue receiving AWS certifications, and we certainly plan to earn the AWS Qualified Software status for other applications. And, we develop new solutions based on cloud-native technology.

You can harness the benefits that the cloud offers only in case you never stop improving your solutions. So, there is still so much work ahead!

We continue to share our experience of switching to and working with Amazon Web Services. Earlier we’ve talked about some challenges and the switching process itself. And since then, we’ve made a lot of progress.

For example, we already have our own AWS guru, who managed to get 12 certificates out of 12. Our AWS community builder, Stanislav Kolenkin, is constantly being certified himself, inspiring the entire team to also move forward steadily.

What else have we made? In spring, we started switching to Terraform. This process is closely related to the company’s global transition to AWS Cloud. Although Terraform is used mainly to work with cloud infrastructure — Amazon, Google, Azure — its functionality is much more extensive. We’ve already tested its tools, and now we share our experience with others.

We also managed to design an AWS-based service for calculating promotions for hundreds of stores and launched a new service, elKasa, which is described below.

The point of the project is to replace hardware Payment Transaction Registers (PTR) with Software Payment Transaction Registers (SPTR) and to administer them from the cloud. To put it simply and more specifically, with AWS capabilities, we managed to improve the cash register service in Silpo supermarkets.

So, what are the advantages of a software PTR?

1. Security and safety of data storage, especially in conditions of volatility due to a full-scale war.

2. Implementation of the option of issuing electronic receipts in accordance with Ukrainian legislation. This also allows to:

• protect the environment: fewer paper receipts mean more preserved forests and less waste;
• reduce the checkout time, which leads to less queuing;
• retain access to the receipt when proceeding with online purchase (e-com);
• optimize the cost of thermal tape and cash register equipment.

3. Automation of reporting and administration of access to the system for employees. Previously, these tasks were mostly manual, time-consuming, and subject to human error.

Nothing’s gonna reSPTRain us: first steps and challenges.

In April 2022, 13 specialists joined together to form a project team. There were .NET/SQL experts, DevOps, automated and manual testers, business analysts, architects, and project managers on the team.

The team took up AWS training and certification a bit later. They started with sorting out all the nuances on their own. The team relied entirely on their experience and knowledge and expanded it vastly in the process. This way of proceeding was very helpful in passing the AWS certification later.

But there were other challenges. In the midst of massive rocket attacks, the project team had to work exclusively online. Sometimes they even managed to work from the subway or other bomb shelters with no electricity, heat, or internet connection. But this did not stop them. Nothing’s gonna reSPTRain us, they used to make fun with each other those days.

Indeed, it was really incredible, we are proud of our colleagues!

The first results

By the end of November 2022, the team managed to launch cloud-based administration for the first SPTRs in Silpo stores.

Today we’ve:

1. Integrated 4126 cash registers.

2. Covered 315 stores by 100% so far and continue to cover newly opened branches.

3. The cloud archive of receipts is already more than 140 million, and we’ve increased it by 25 million over the past month.

The team has also provided the colleagues with all the development, and they have already started synchronizing the app for Guests and elKasa. Now, our guests can choose to get electronic receipts instead of paper ones in most of our stores.

We have ambitious goals, so the work and rollout go on.

We intend not to rest on our laurels, but to push through our limits because AWS capabilities provide unlimited space for cool solutions.

This could have been the introduction to a standard Terraform use article. However, our focus today is on the configuration and settings of our development tools.

While most people utilize Terraform for managing cloud infrastructure on platforms like Amazon, Google, and Azure, this tool offers a broader range of functionality. It allows us to manage third-party services through a substantial selection of providers, both official and community-based. We employ Terraform as a comprehensive tool for managing the infrastructure we work with, including GitLab, SonarQube, and potentially Artifactory in the future. These tools serve as the foundation for team workspace organization.

How We Facilitate New Project Onboarding

Consider a scenario where a new project within the company needs the creation of GitLab subgroups, repositories, and related actions. As we also employ SonarQube for code analysis, we replicate similar actions for a new project on SonarQube, ensuring seamless code quality testing integration for teams. Terraform generates a group for the project’s repositories in GitLab, adhering to our naming conventions. Additionally, we create security groups for communication with Active Directory, which simplifies access to GitLab projects for our developers.

To streamline the integration of these two tools, we’ve automated the process. In the traditional manual approach, users need to add links, integration tokens, or specific tasks to the pipeline. Our approach simplifies this by first creating entities on GitLab and SonarQube and then adding integration tokens to the desired GitLab project.

Once this is completed, the team can commence work with automation at the forefront. We’ve also implemented automatic token rotation, enhancing security by automatically regenerating tokens in case of a compromise. This eliminates key management issues and bolsters security.

Artifactory can be seamlessly integrated into this pipeline, providing a platform for private repositories of various technologies, including containers and Terraform modules. Typically, the outcome of an activity is an artifact, making separate repositories for these artifacts a best practice. When a new project is initiated, Artifactory automatically creates a space for teams to upload their work results.

The Engineer’s Workflow

The Terraform code configuration with SonarQube is illustrated in the provided example. This demonstrates the key advantages of Terraform.

```

#################################################################################

# SonarQube project configuration

#################################################################################

resource “sonarqube_project” “this” {

name = var.name

project = var.name

visibility = var.project_visability_level

tags = [lower(var.project_tag)]

}

resource “sonarqube_permissions” “this” {

group_name = “temabit/${lower(var.project_tag)}”

project_key = sonarqube_project.this.project

permissions = [“codeviewer”, “user”]

}

#################################################################################

# SonarQube Project Analysis token

#################################################################################

resource “terraform_data” “retention_trigger” {

input = var.project_token_retention

triggers_replace = true

}

resource “sonarqube_user_token” “this” {

name = “${lower(var.project_tag)}-${var.name}-project-token”

type = var.token_access_level

project_key = sonarqube_project.this.project

expiration_date = local.user_token_expires_at

login_name = var.sonarqube_integration_user

lifecycle {

ignore_changes = [

expiration_date

]

replace_triggered_by = [

terraform_data.retention_trigger

]

}

}

#################################################################################

# Gitlab Project SonarQube token

#################################################################################

data “gitlab_project” “this” {

path_with_namespace = var.project_subgroup == null ? “temabit/${var.project_tag}/${var.name}” : “temabit/${var.project_tag}/${var.project_subgroup}/${var.name}”

}

resource “gitlab_project_variable” “sonar_token” {

project = data.gitlab_project.this.id

key = “SONAR_TOKEN”

value = sonarqube_user_token.this.token

masked = true

}

resource “gitlab_project_variable” “sonar_url” {

project = data.gitlab_project.this.id

key = “SONAR_HOST_URL”

value = var.sonarqube_url

masked = false

}

```

Key Terraform Advantages

One of the significant benefits of our process is the streamlined pipeline. The process is highly efficient. We initiate a ticket, assign it a name, and our engineer configures Terraform, ensuring everything functions correctly. Once the pipeline is approved, we obtain all the necessary settings for working on a new project within minutes. The ticket processing time stands at 30 minutes, although our future plans include full automation of this process.

Why do we follow this approach? It significantly reduces the time engineers spend on request processing and minimizes routine tasks. It allows engineers to concentrate their focus without distractions. Everything is monitored, ensuring visibility into all changes.

Adapting to Terraform as a Team

We essentially began anew with GitLab as our primary tool, embarking on comprehensive automation alongside our experienced team. While our team possessed prior Terraform experience, our approach to using Terraform evolved, broadening our horizons and enhancing our technical expertise. For instance, we explore the potential benefits of the AWS CDK, but that’s a topic for another article.

Our efforts have yielded positive results, reducing the onboarding process from three hours to just 15 minutes. Of course, human factors may come into play, such as an engineer’s absence from the workplace, which may need additional time. However, we are actively exploring solutions to minimize such factors, striving to find the optimal set of tools. The main goal remains the substantial reduction of time spent on technical and routine tasks.

We will continue sharing updates — stay tuned.

Oleksandr Sapozhnikov, Lead of SRE Team, TemaBit Fozzy Group

We’ve said it before, our team is out of this world. And we have some incredible news because we’re migrating to the cloud.

Now, in TemaBit Fozzy Group, we have our own AWS guru who managed to get all 12 AWS certificates. Our AWS community builder, Stanislav Kolenkin, drives the team, cloud transformation, and became the team’s certification driver. He achieved all the possible AWS certificates available today. Congratulations, we’re so proud of it!

How did he do it? And what were the obstacles and motivation?

What’s the most challenging part of certification? Taking the first step and overcoming laziness. So, if you have a goal, get inspired by the example and start with the basic level.

Attend webinars, learn, practice, and get certified. It’s important to move gradually and start from the foundation. It’s AWS Certified Solutions Architect Associate since Amazon has a well-thought-out learning path. And if you take the exams step by step, questions and materials come from previous certifications.

Specialized levels are quite challenging. The exam takes three hours, and the answer choices are very similar. The questions focus both on theory and practice, so don’t ignore them during your training. It’s also worth starting from the basic level to practice similar tests.

By the way, it took Stanislav a little over a year to complete the certification journey. And this is not to mention the other certificates he managed to obtain during this time. He prepared for each one for two weeks to a month in the evenings after work. So, everything is possible.

On support and mentorship

Our Cloud Foundation, which is involved in cloud migration and cloud transformation and wants to grow in that direction, has the opportunity to join the training, take courses, and attend webinars. At every stage of certification, our colleagues are not left alone. Experienced mentors under Stanislav’s leadership guide their journey, share their experiences, and provide useful advice during the certification journey.

Why do you need certificates?

Certifications are not a magic key to all doors. You should polish your new knowledge in practice because theory alone won’t be enough for the job. Passing a certification exam with brain dumps is noticeable right at the beginning of an interview. It negatively affects the hiring process. So, you should only take certification exams with a good understanding of the subject. Be sure to work on practical applications for each certification topic.

In some countries, companies need you to prove your knowledge. They won’t hire even a very strong expert without certificates. This applies to some Israeli or American companies. So, don’t miss your chance.

And it’s also a good idea to keep an eye on trends and new Amazon patterns because they don’t stand still. For example, while we were writing this article, AWS added a 13th certificate, AWS Certified Data Engineer — Associate Beta. If you’ve already joined the certification process, you’ll start listening to webinars and courses. So, it’s another opportunity to boost your skills in innovation. By the way, you can practice with sample exam questions and attend training sessions here. Always remember about practice. Through practice, you can gain the necessary knowledge and get the right confirmation.

Impossible is nothing. The key thing is to keep working and achieving your goals.

https://pages.awscloud.com/rs/112-TZM-766/images/T&C-Overview-Brochure_Mar-2020_digital-2020.pdf

Stanislav Kolenkin's profile on Credly

Stanislav Kolenkin, Head of Cloud Foundation at TemaBit Fozzy Group

Fozzy Group переходить на AWS: перші кроки та виклики

З початком повномасштабної війни росії проти України Fozzy Group зіштовхнулася з новими викликами. Для наших бізнесів критично важливо було забезпечити українців продуктами першої необхідності, швидко адаптуватися до змін та побудувати нові логістичні ланцюжки.

ІТ інфраструктура TemaBit Fozzy Group відіграла одну з ключових ролей у вирішенні цих завдань. Завдяки їй компанія могла:

• швидко реагувати на зміни попиту на продукти;
• налагодити постачання товарів з нових джерел;
• підтримувати роботу магазинів та складів.

Також це був рік, коли ми планували оновити обладнання центрів обробки даних (ЦОД), яке уже ставало застарілим. Будь-які розстрочки та великі капіталовкладення були недоступними. Крім того, час доставки обладнання значно збільшився. Так, замість 90 днів, які ми зазвичай чекали на нове обладнання, довелося б чекати удвічі, а іноді й втричі довше. Це вже не кажучи про випадки, коли взагалі було неможливо передбачити терміни доставки.
З початком повномасштабної війни Fozzy Group опинилася перед викликами. Зберігати дані на серверах на землі було ризиковано, а оновити обладнання ЦОД було непросто. Тому було прийнято рішення рухатися у хмару. Це дозволило компанії продовжувати роботу та забезпечувати українців продуктами першої необхідності.

Чому ми обрали AWS?

Проаналізувавши різні варіанти клаудів наш вибір впав на AWS, оскільки їхня політика партнерської співпраці спонукає до розвитку експертної кваліфікації IT-команд за напрямком cloud strategy.

Також AWS дає можливості оптимізувати хмарні ресурси та детально спостерігати за ними у хмарі. Одна з найважливіших хмарних вигід — можливість платити тільки за ті ресурси, які ви фактично використовуєте, та не переплачувати наперед. Не треба інвестувати або гадати, скільки треба вкласти грошей — Pay as you Go.

З початку співпраці наша компанія збагатила внутрішню експертизу і вже налічує 15+ AWS Certified Solutions Architect (Associate and Professional) та інші сертифікації PRO and Specialty рівнів. На цьому ми не зупиняємося та продовжуємо сертифікацію наступних. Цей процес драйвить Станіслав Коленкін. У його команді майже кожен пройшов успішне навчання та має мінімум 2 сертифікати включно з AWS Solution Architect Associate.

Серед наших колег в TemaBit за перше півріччя 82 людини пройшли навчання із роботи з AWS сервісами на різні тематики: робота в хмарі, безпека в хмарі, створення архітектурних рішень для безперебійного забезпечення роботи компанії.

Jornеy map: де ми є та куди прямуємо

Підготовка

Уже на початку ми розуміли, що перехід у хмару має бути оціненим з усіх сторін та ретельно проаналізованим. Адже необдумане рішення може призвести до безлічі ризиків.

Аудит

Першим кроком ми зробили великий аудит інфраструктури, знайшли всі системи та пішли до бізнесів, щоб визначити ступінь їхньої критичності. А бізнесів у нас досить багато, серед ключових — мережі «Сільпо» (300+ магазинів), «Фора» (300+ магазинів), «Траш» (90+ магазинів), гіпермаркети Fozzy C&C (8 магазинів), мережа аптек «Біла ромашка» (90+ аптеки), магазини техніки Ringoo, e-com (доставка “Сільпо” та LOKO), Власна Логістика, маркетплейс MAUDAU, магазини зоотоварів E-ZOO та ін.

Тут нас чекав виклик, адже оцінити критичність своїх систем бізнеси не могли, спершу все здавалося надважливим. Та коли дійшли до фінансової складової і постало питання, чи може бізнес, скажімо, платити 100 тисяч доларів на рік за сервіс, почали задумуватися. Щоб визначити критичність та черговість перенесення сервісів у хмару, ми розробили алгоритм, який запропонували заповнити кожному з бізнесів.

Спершу ми визначили розподіл між собою системи за хвилями міграції. Ставало питання: які системи потрібно нести в хмару вже сьогодні, які можуть почекати, наприклад, два-три місяці, а які й взагалі можна відкласти на кінець року. Для визначення цієї черговості були сформовані певні критерії, які й формували хвилі міграції та включали перелік систем. В нашому випадку ми отримали 6 хвиль міграції системи в хмару. Визначення та погодження хвиль видались для нас складною та тривалою аналітичною роботою.

Ті ж системи, які бізнеси віднесли до некритичних, ми залишили у ЦОДі та захистили реплікою. Наразі копії всіх віртуальних машин є в AWS. Та, нажаль, ніхто не може з впевненістю сказати, скільки зусиль та часу займе відновлення цих систем у разі їх втрати.

Team composition

Ми сформували команду фахівців, яка буде займається міграцією в AWS. Були сформовані декілька повнофункціональних команд міграції, кожна з яких має у своєму складі PM, архітектора, системних інженерів, DBA. Команди можуть працювати паралельно для пришвидшення міграції.

Ми починали міграцію з lift and shift, тобто без зміни технологій. Тому на початку для більшості працівників, наприклад, розробників, у процесах нічого не змінювалося. Як вони працювали з нашим ЦОДом, так вони й продовжили працювати з AWS, для них це було знайоме середовище.

Коли вже почали перехід на нативні сервіси AWS, потрібно було розуміти, наприклад, як організовувати процес бекапу, мережу, захист та ін. Найважчим тут було паралельно з першим процесом здобувати необхідні знання для подальшої роботи.

Вже для систем, які рефакторяться, потрібна специфічна експертиза і тут ми отримували підтримку від фахівців з AWS. Паралельно наші працівники проходили навчання та сертифікацію в AWS. Також ми посилили нашу команду досвідченими фахівцями, які мають великий досвід у міграції в інших великих компаніях.

Варто зазначити, що на всіх етапах міграції нас підтримує команда експертів з AWS та нова cloud Foundation команда зі скілованими спеціалістами, у яких за спиною сотні успішних проєктів з міграції та AWS.

Міграція

Основним драйвером комбінованого підходу, який ми обрали, — є війна проти України. Нам дуже важливо якнайшвидше убезпечити ключові дані, які необхідні для безперебійної роботи бізнесу.

Підходів до переходу є досить багато: Rehost, Replatform, Refactor, Retire, Retain, Repurchase.

Ми ж для себе виділили такі два:

1. Перенесення даних, як є — lift and shift;
2. Перенесення з реархітектурою, щоб дані ставали native cloud.

Ми обрали для Fozzy Group комбінований шлях, тобто рухаємося одночасно двома шляхами. Ми не могли зупинити ЦОД на тривалий період, адже це як робити операцію на живому серці. Робота вимагає філігранності та обережності. Тому ми домовлялися з бізнесами, і частинами переносили системи, не зупиняючи роботу магазинів.

Для Critical Application робимо реархітектуру і далі вони йдуть на cloud native application. Інші додатки переносимо «як є», а далі вже поступово будемо робити їх cloud native.

До кінця року плануємо закінчити міграцію Critical Application з реархітектурою та паралельно перенести важливі дані «як є».

Далі, безумовно, будемо працювати з cloud native. Також, у нас є дані, які не можна перенести у cloud, навіть з підходом «як є», тому їх треба рефакторити, а це вже, по суті, новий проєкт, який також буде братися до роботи.

Слід зауважити, що всі нові розробки відразу реалізуються у клауді на базі AWS. Таких проєктів стає дедалі більше й управління ними потребує все більшого ресурсу з точки зору фахівців. Тому тут ми вирішили рухатися шляхом Platform Engineering і в нас з’явилася Platform team та SRE у рамках cloud Foundation.

Cloud Foundation — це інтегрована платформа для ефективного створення, управління та оптимізації хмарної інфраструктури. Вона вибудовує автоматизовані патерни, завдяки яким реалізувати та управляти проєктами стає легше, швидше та з меншим залученням фахівців. Тобто робота над запуском нового проєкту стає більш автоматизованою та імплементується підхід Infrastructure as code (IaC).

Поради бізнесам, які задумалися про міграцію в AWS

• Для переходу в AWS варто зважено оцінювати та прогнозувати витрати. AWS пропонує широкий спектр послуг, кожна з яких має власну цінову політику. Недооцінка може призвести до несподіваних високих витрат, тоді як правильна оцінка допоможе ефективно використовувати ресурси та уникнути фінансових збитків. Важливо ретельно проаналізувати навантаження та спрогнозувати витрати на розміщення в AWS не тільки на перший, але й на другий та третій роки, бо ці витрати можуть стрімко зростати, адже після міграції настає фаза оптимізації ресурсів, right sizing, reservation.
• Перехід компанії на AWS вимагає часу та фахових знань для налагодження та управління інфраструктурою. Для деяких компаній це може стати викликом, особливо якщо вони не мають внутрішніх ресурсів для такого роду роботи. Системи можуть почати поводитися неочікувано під час покрокового перенесення систем в хмару без детального їх опису та залежностей одна від одної. Це може призвести до повної чи часткової відмови частини систем, які залежать від тої, що мігрується.
• Під час переходу на AWS важливо застосовувати IaC підхід і заборонити clickOps.
• Варто одразу прорахувати та вивільнити необхідну кількість людей для всіх процесів, оскільки без цього строки виконання міграції можуть порушитися.
• Під час переходу зі старих процесів на нові хмарні змінюється і майндсет у людей, тобто відбувається певна трансформація. Тому тут важливо мати команду сильних DevOps фахівців, які допоможуть колегам із різних підрозділів справитися із культуральними змінами та зроблять процес трансформації безшовним, а також дозволять оптимізувати фінансові і людські витрати.
• Почати впроваджувати FinOps фреймворк та культуру.

Замість висновків

Перехід компанії на AWS може мати багато переваг, таких як масштабованість, гнучкість та широка функціональність. Однак, слід враховувати потребу в фахових знаннях для ефективного управління.

Перед прийняттям рішення про перехід на AWS, варто ретельно проаналізувати свої потреби та можливості, а також порівняти його з іншими схожими сервісами. А далі — багато цікавої та складної роботи, зміна майндсету компанії та постійне навчання, аби ставати ще потужнішими.

І пам’ятайте, AWS — це не той звір, який кусається, до нього просто треба знайти підхід.