Multi-Cloud Banking: Moving a GCC Bank to Kubernetes on Two Clouds

Oracle Cloud as the primary site, Huawei Cloud as the secondary, both serving traffic, and the data never leaving the country

Part 1 of the Multi-Cloud Banking series.

Most banks have a disaster recovery site. Very few have ever sent real customer traffic to it. The DR data centre sits there, kept roughly in sync, tested once a year in a carefully staged exercise, and everyone quietly hopes the day it’s really needed never comes. When it does, the runbook is out of date, a firewall rule is missing, and the 30-minute recovery target turns into a long night.

This series is about how we replaced that model at a GCC bank. I joined the programme as a consultant on the platform side. The goal was to move the bank’s entire infrastructure, more than a hundred business applications, off legacy hosting and onto Kubernetes within two years, across two clouds from two different vendors, both serving traffic. Oracle Cloud is the primary site, Huawei Cloud the secondary, and the bank’s data never leaves the country.

This first part covers the why and the overall architecture. The rest of the series goes layer by layer.

Why leave the data centre at all

Before this programme, the bank ran its infrastructure the way most banks do. It bought servers and storage arrays every few years, racked them, patched their firmware, renewed support contracts with a long list of hardware vendors, and planned capacity a year ahead because a new storage array took months to arrive. A large share of the infrastructure team’s time went into keeping that estate alive, not into the applications the bank actually earns money from.

Moving to the cloud changes what the bank operates.

• The provider runs the infrastructure. Hardware, storage, the network fabric, the virtualisation layer and the cloud’s own control plane are the provider’s job, under a contract with agreed service levels. Failed disks, firmware upgrades and hardware refreshes stop being the bank’s problem.
• Capacity is a request, not a purchase. A dedicated region still has a finite size, and growing it is planned with the provider. But a new environment, cluster or database comes out of that capacity in hours, through Terraform, instead of after a procurement cycle.
• A running cost instead of capital purchases. The old model was capex: a large hardware purchase every few years, depreciated over its life, and sized for growth that might never come. The new one is opex: a service fee for infrastructure the provider refreshes. A dedicated region isn’t pay-as-you-go, though. It’s sold as a multi-year commitment, so what the bank gets is a predictable running cost in place of irregular capital spikes, not unlimited elasticity.
• Everything is an API. In the old estate, a new application meant separate tickets for virtual machines, storage, firewall rules and DNS, each handled by a different team. In the cloud, all of that is code, reviewed in a merge request and repeatable for the next environment (Part 3). It’s better evidence for auditors too: what’s in Git is what’s running.
• A second site without a second build. A second data centre built by the bank would have doubled the hardware, the facilities and the people to run them. A second dedicated region from another vendor gives the bank a second site that someone else operates.
• The team moves up the stack. Engineers who spent their days on hardware and hypervisors now work on the platform, automation and reliability, the work that decides whether mobile banking is up. Part 8 covers how the team made that move.

The bank’s own data centres don’t disappear. They stay the network hub (Part 4), home to the internet edge and to the links to branches, ATMs and partners. What they stop being is the place where the bank runs its servers.

Leaving VMware and the licence bill behind

Running its own hardware wasn’t the only cost the bank wanted to leave behind. The legacy estate ran on VMware, and like many banks, this one had watched its renewals change shape after Broadcom bought VMware: perpetual licences gone, products sold only as subscription bundles, and a bigger bill for the same servers. Licences the bank had bought once as capital spending turned into a yearly operating cost anyway, just a larger one. On top of the hypervisor sat the usual layers of commercial middleware and databases, each priced per core or per instance, each with its own audit clause.

The new platform drops most of that. The bank no longer licenses a hypervisor: both clouds run on the providers’ own compute, and workloads run on Kubernetes above it. Above the infrastructure, everything is open source, with an enterprise support contract where the bank wants one. That changes more than the invoice.

• The bank pays for support, not for the right to run. Adding a node, a replica or a whole environment doesn’t trigger a licence true-up, and there’s no core count to argue about in an audit. Scaling up for month-end is a capacity question, not a procurement one.
• Test environments cost hardware, not licences. The shared non-production cluster and the staging pair in Part 3 run exactly the software production runs. Under per-core licensing, every one of those copies would have been a line in the budget, and staging would have been cut down to save money.
• The support vendor can be replaced. If a support partner underperforms or raises its prices, the bank can move to another one, or support a component itself, without changing the platform.
• The same software runs on both clouds. An Oracle managed service doesn’t exist on Huawei, and the other way round. PostgreSQL, Kafka, Istio and Vault run the same way on both, which is what makes a two-vendor design possible at all.

A few components don’t meet the strict definition, and those calls were made knowingly. MongoDB is published under the SSPL and Elastic Security under the Elastic License: both are source-available rather than open source. Their terms restrict offering the software as a service to others, which the bank doesn’t do, and paid subscriptions cover the support and features it needs. Portworx, used for a handful of legacy volumes (Part 6), is a commercial product. All three sit on a short list of exceptions that’s reviewed whenever a vendor changes its licence terms, which in recent years several have.

Open source isn’t free. Support subscriptions cost money, and the bank needs engineers who can run Kafka and PostgreSQL when the support line is slow to answer. Oracle Database stays licensed for core banking, and the two clouds are commercial contracts in their own right. The difference is where the money goes: into infrastructure and support the bank chose, instead of licences that grow every time it scales.

The constraints we started from

A bank doesn’t get to pick its architecture from a blank page. Four constraints shaped every decision that followed.

• Data residency. The central bank requires customer data to stay inside the country. That rules out the obvious answer, a public hyperscaler region next door, for anything that touches customer data. Whatever we built had to run on infrastructure physically located in-country.
• No single vendor to depend on. Regulators in the region increasingly look at concentration risk: what happens to the bank if its one cloud provider has a bad day, a contract dispute, or a sanctions problem. The bank wanted a second site that fails differently from the first, which means a different vendor, not just a different building.
• No lock-in above the infrastructure. Every component had to be open source with enterprise support available, so the bank could switch support vendors or support itself. Proprietary managed services were acceptable for the infrastructure layer only. This one turned out to pay for a lot of the programme, as the section on leaving VMware above explains.
• Recovery targets set by the business. The bank’s service-tiering framework gives the most critical services (core banking, mobile and internet banking, payments) a 99.9% availability target, a recovery time of 30 minutes or less, and at most 5 minutes of data loss. A cold DR site can’t meet that reliably. A warm one, exercised once a year, barely can.

Put together, those constraints point at one shape: two in-country sites, from two vendors, both running production all the time.

Two dedicated cloud regions, not two data centres

Neither of the bank’s cloud providers runs a public region in the country, so both sites are dedicated regions: the provider’s cloud, installed in a data centre in the country and run for the customer.

• The primary site is an Oracle Cloud dedicated region (OCI Dedicated Region Cloud@Customer). It’s a full OCI region with the same APIs, services and console as the public cloud, but the hardware sits in-country. Kubernetes runs on OKE, Oracle’s managed Kubernetes service.
• The secondary site is a dedicated Huawei Cloud region (Huawei Cloud Stack) in a second city, roughly 500 km away. Kubernetes runs on CCE, Huawei’s managed Kubernetes service.

Why Oracle as the primary? The bank’s core banking system and much of its estate already ran on Oracle databases. Keeping the system of record next to the Oracle-native tooling it depends on, and the licences that come with it, made OCI the natural home for the heaviest, most stateful workloads. Huawei brought a second, fully independent cloud stack: different hardware, different control plane, different operations team. A bug in one provider’s load balancer or IAM can’t take both sites down at once.

The distance matters too. About 500 km is far enough that a regional power or network event is unlikely to hit both cities. It’s also close enough that the round trip between sites is a handful of milliseconds. That’s what makes synchronous replication of the core banking database affordable, as Part 6 explains.

Kubernetes is the common layer

Running two clouds from two vendors only works if you stop caring which cloud you’re on as early as possible. For us, that line is Kubernetes.

Below Kubernetes, everything is cloud-specific, and we accepted that. Each site has its own Terraform, written in layers against its own provider: the environment, the compartment or enterprise project, DNS, the network, the managed cluster, cluster add-ons, object storage. OCI compartments and Huawei enterprise projects aren’t the same thing, and pretending otherwise with a leaky abstraction would have cost more than it saved.

Get Ayoub Reda’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Above Kubernetes, everything is identical. Both clusters run the same platform, delivered by the same GitOps chart, with only a values overlay per cluster:

• Argo CD in a hub-and-spoke setup, deploying both clusters from one place, outside both of them (more on that below).
• Istio as the service mesh, joining both clusters into one mesh.
• Cilium for pod networking policy and flow visibility.
• Vault and External Secrets for secrets, cert-manager for certificates, and Kyverno for admission policy.
• Kubernetes operators for every stateful service: CloudNativePG for PostgreSQL, Strimzi for Kafka, and operators for MongoDB, RabbitMQ, Redis and Elasticsearch.

An application team ships one Helm chart and one Argo CD application, and it lands on both clusters the same way. They don’t write anything that knows whether it’s running on OKE or on CCE. Part 3 covers that platform in detail.

Cloud-specific below Kubernetes, identical above it.

What “active-active” means here

“Active-active” gets used loosely, so here is what it covers in this design and what it doesn’t.

Stateless services run active-active. Every API, every channel back-end and every integration service runs in both clusters at all times, with real traffic. Istio joins the two clusters into one mesh using the multi-primary, multi-network model: each cloud has its own Istio control plane, and traffic between the clouds crosses through dedicated east-west gateways with mutual TLS end to end. A service calls its dependencies locally first. When a local dependency fails or its endpoints are ejected for errors, Istio sends the call to the healthy copy in the other cloud, without the caller doing anything. Part 5 covers how that works and how to stop a failover from turning into a retry storm.

Oracle is the preferred site, not the only one. Customer traffic enters through a global load balancer that sends most of it to the Oracle site and some of it to Huawei. Both sites take real traffic every day, so the secondary is always proven to work. That’s the opposite of a DR site that sits idle until the worst day of the year.

Data has one writer at a time. Writing the same account balance in two places at once and reconciling it later is not something a bank should do. Each database has a single primary, normally at the Oracle site, with a standby at the Huawei site. The core banking database, on Oracle Database outside Kubernetes, replicates synchronously. Everything else replicates asynchronously, with its lag measured against its tier’s data-loss limit. Part 6 is about state, service by service.

So “active-active” in this series means: both sites serve traffic all the time, any stateless service can fail over to the other cloud on its own, and data fails over in a controlled way within the recovery targets of its tier.

The architecture at a glance

Two dedicated cloud regions joined by the bank’s network and one Istio mesh, run and observed from a third cloud.

Some of that picture needs explaining.

The bank’s network is the hub. Neither cloud connects to the other directly. Each one connects to the bank’s own network over redundant private circuits, and traffic between the clouds transits the bank’s network. The bank already owns that network, along with its firewalls, monitoring and the branches, ATMs and partner links that live on it. Part 4 covers the circuits, the encryption on top of them, and DNS across both sites.

The address plan has no overlaps. Each cloud has its own address range for nodes and separate ranges for pods and Kubernetes services, and none of them overlap with each other or with the bank’s existing network. That sounds obvious, but it’s the one decision you can’t change later. It’s also what lets a database in Huawei replicate from a database in Oracle without network address translation in the middle.

Nothing is exposed directly. Both clusters use internal load balancers only. Customer traffic comes in through the bank’s existing edge, which handles the internet-facing side and global load balancing, and is forwarded to the Istio ingress gateway in each cluster.

Control and observability live in a third place. If the tooling for a two-site system runs in one of those two sites, losing that site also takes away your ability to see what happened and to act on the site that’s left. So both live on AWS, in an in-country Local Zone: outside both clouds and inside the country. It’s the bank’s management site.

• The platform control plane: GitLab, its CI runners, Harbor as the source of every image, and the Argo CD hub that deploys both clusters. If Oracle goes down, we can still change, scale and fix Huawei.
• A Grafana LGTM stack: Mimir for metrics, Loki for logs, Tempo for traces, Grafana on top. Part 7 covers how that pipeline works and what we had to strip from the telemetry before it could leave the clusters.

What we kept off the management site matters just as much. Nothing a running pod needs at runtime lives there. Each site runs its own Vault and its own registry mirror, so a pod can restart, scale or move between nodes with AWS unreachable. Losing the management site freezes changes; it doesn’t stop traffic.

Tiering decides how much of this each service gets

Not every service needs everything in this picture, and paying for a synchronous standby on a marketing website would be wasteful. The bank’s tiering framework decides who gets what.

Every business service is scored on three separate axes, confidentiality, integrity and availability, each from tier 1 (critical) to tier 4 (basic). Applications inherit the highest tier of every business service they support, and the infrastructure under them inherits from the applications. A rating like C1-I1-A1 (core banking, mobile banking) means tier 1 on all three. C4-I3-A2 (a public website) means very different controls.

Those tiers translate directly into the architecture:

• Availability tier decides the replica counts in each cluster, whether a service must run in both clouds, and how aggressively the mesh fails it over.
• Integrity tier decides how much data loss a dataset can tolerate, and from that, how it replicates, how often backups run and how long they’re kept.
• Confidentiality tier decides network segmentation, encryption and who can access the service at all.

The framework is simple, and few things saved us more time. Arguments like “my service is important” became a lookup table. Part 2 goes through it, alongside the applications a bank actually runs and how each kind lands on this platform.

What we gave up

This architecture has real costs.

• Two of everything to operate. Two Terraform codebases, two cloud consoles, two support contracts, two sets of quotas and two upgrade calendars. OKE and CCE don’t release Kubernetes versions on the same schedule, so keeping both clusters on compatible versions is a continuous job, not a one-off.
• The lowest common denominator above Kubernetes. We couldn’t use OCI’s or Huawei’s managed databases, queues or caches for anything that had to fail over across clouds, because the other cloud doesn’t have the same service. Everything stateful runs on Kubernetes operators, which means the bank runs its own Kafka, PostgreSQL and MongoDB, with enterprise support contracts behind them.
• Cross-cloud networking is the hardest part. Every cross-site call depends on the circuits, the firewalls in the bank’s network, and the east-west gateways. A misconfigured route there looks exactly like an application outage.
• The bill arrives every month. Opex is visible and grows with use, so cost needs managing all the time, not approving once every five years. Two dedicated regions, with tier 1 services sized to carry the whole load at either site (Part 5), are a standing cost the bank pays whether or not a failover ever happens.
• Skills. The team has to know two clouds well enough to troubleshoot both under pressure, and it started with no cloud experience at all. Consultants built the platform alongside the bank’s engineers and handed it over wave by wave; Part 8 covers how.

We accepted those costs because the alternative, a single cloud with a DR site we’d rarely use, would have failed the regulator’s concentration test and the bank’s own recovery targets.

Next

The next part steps back from the infrastructure to what runs on it: the applications a bank actually operates, from core banking to payments and cards, how the tiering framework classifies them, and which ones moved first.

Multi-Cloud Banking

1. Multi-Cloud Banking: Moving a GCC Bank to Kubernetes on Two Clouds (you are here)
2. Multi-Cloud Banking: What a Bank Runs, and How Tiering Decides Where It Lands (coming soon)
3. Multi-Cloud Banking: One Platform on Two Clouds with Terraform and GitOps (coming soon)
4. Multi-Cloud Banking: The Network Between Two Clouds (coming soon)
5. Multi-Cloud Banking: Istio Active-Active Across Two Clouds (coming soon)
6. Multi-Cloud Banking: State Is the Hard Part (coming soon)
7. Multi-Cloud Banking: Observability From a Third Cloud (coming soon)
8. Multi-Cloud Banking: Moving 100+ Applications Onto a Greenfield Platform (coming soon)