Istio- MultiCluster MultiPrimary MultiNetwork Service Mesh on AWS EKS

A practical walkthrough of setting up cross-cluster service mesh with centralized observability using Istio, Prometheus federation, and Centralized Kiali. With Istio’s multi-cluster capabilities, you can create a unified service mesh that treats these geographically separated clusters as one logical network. Four pillars of Istio are

1. Traffic Management

Istio gives operators fine-grained control over the flow of traffic and API calls between services. It offloads complex routing and load balancing from the application to the mesh.

2. Security

Istio provides a robust security framework for implementing zero-trust networking without modifying the application code.

3. Observability

Istio generates deep telemetry for all traffic flowing through the mesh, providing actionable insights into network behavior and service health.

4. Extensibility

Istio’s architecture is designed to integrate easily with organizational policies, telemetry backends, and custom filtering.

This blog is created in 06/07/2026. Please revisit for updates or comment if you have any queries.

Istio comes with mTLS by default. It is important that Istio shares same CA certs with other clusters that it connects with. It enables the traffic to be encrypted between clusters. In this section, I am not going to enable CA certificates and security for cost reduction and make sure I am within AWS free tier limit.

Multi Primary Clusters on Separate Networks

Prerequisites:

1. An AWS Account with two VPC’s(app-vpc, platform-vpc) running in ap-south-1 region.
2. Terraform- Deploy the AWS Resources through terraform.
3. istioctl CLI tool to install istio inside the clusters.
4. Basic understanding of Kubernetes and ServiceMesh.
5. kubectl- To connect to kubernetes cluster and run commands against the cluster.

Architecture

Architecture of my service mesh

Repository and Folder Structure

Github Link: https://github.com/sydsulai/centralized-istio-platform

centralized-platform/
├── README.md                          # Setup guide and issue tracking
│
├── centralized-cluster/               # Platform cluster (central hub)
│   ├── cluster-configuration/
│   │   ├── addons_manifests.sh       # Installation script/manual trigger
│   │   ├── common-manifests/          # Cluster wide common manifests
│   │   └── service-mesh-config/      # Cluster wide service mesh configuration
│   └── infrastructure/                # Terraform for platform cluster
│
├── app-cluster-1/                     # Application cluster (remote)
│   ├── cluster-configuration/
│   │   ├── addons_manifests.sh
│   │   ├── common-manifests/          # Cluster wide common manifests
│   │   └── service-mesh-config/        # Cluster wide service mesh configuration
│   └── infrastructure/                # Terraform for app cluster
│
├── customized-container-images/       # Custom Docker images
│   ├── Dockerfile.aws-xray           # AWS X-Ray sidecar
│   └── envoy/
│       ├── Dockerfile.envoy          # Custom Envoy build
│       └── config.yaml               # Envoy configuration
│
└── iam-policies-roles/                # Shared IAM policies and roles
    ├── iam.tf, iam_policy.json       # IAM definitions for both clusters
    └── dev.auto.tfvars

Steps to Create Multi-Primary Multi-Network Service Mesh

Step1: Update the kubeconfig in your local

export PLATFORM_CLUSTER=$(kubectl config get-contexts -o name | grep -i platform-cluster)
export APP_CLUSTER=$(kubectl config get-contexts -o name | grep -i app-cluster-01)

Step2: Deploy aws-load-balancer-controller in both clusters(Optional for application reachability)

helm repo add eks https://aws.github.io/eks-charts

helm repo update eks

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=<cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=ap-south-1 \
  --set vpcId=<vpc-id> \
  --set image.repository=public.ecr.aws/eks/aws-load-balancer-controller

Step3: Label the namespace(Both clusters)

Istio identifies by the namespace label.

kubectl label namespace <namespace> istio.io/rev=default --overwrite

Step4: Install Istio(Both Clusters)

There are 2 sections in the below yaml, Store it separately based on the cluster config.

# App Cluster IstioConfig

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: eastwest
spec:
  revision: "default"
  profile: demo
  meshConfig:
    defaultProviders:
      metrics:
      - prometheus
    defaultConfig:
      proxyStatsMatcher:
        inclusionRegexps:
        - .*
  components:
    ingressGateways:
      - name: istio-ingressgateway
        label: 
          istio: ingressgateway
          app: istio-ingressgateway
        enabled: true
      - name: istio-eastwestgateway
        label:
          istio: eastwestgateway
          app: istio-eastwestgateway
          topology.istio.io/network: app-vpc
        enabled: true
        k8s:
          env:
            # traffic through this gateway should be routed inside the network
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: app-vpc
          service:
            ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: tls
                port: 15443
                targetPort: 15443
              - name: tls-istiod
                port: 15012
                targetPort: 15012
              - name: tls-webhook
                port: 15017
                targetPort: 15017
  values:
    gateways:
      istio-ingressgateway:
        injectionTemplate: gateway
    global:
      meshID: mesh1
      multiCluster:
        clusterName: app-cluster-01
      network: app-vpc

---
# Platform Cluster Config
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: eastwest
spec:
  revision: "default"
  profile: demo
  meshConfig:
    defaultProviders:
      metrics:
      - prometheus
    defaultConfig:
      proxyStatsMatcher:
        inclusionRegexps:
        - .*
  components:
    ingressGateways:
      - name: istio-ingressgateway
        label: 
          istio: ingressgateway
          app: istio-ingressgateway
        enabled: true
      - name: istio-eastwestgateway
        label:
          istio: eastwestgateway
          app: istio-eastwestgateway
          topology.istio.io/network: platform-vpc
        enabled: true
        k8s:
          env:
            # traffic through this gateway should be routed inside the network
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: platform-vpc
          service:
            ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: tls
                port: 15443
                targetPort: 15443
              - name: tls-istiod
                port: 15012
                targetPort: 15012
              - name: tls-webhook
                port: 15017
                targetPort: 15017
  values:
    gateways:
      istio-ingressgateway:
        injectionTemplate: gateway
    global:
      meshID: mesh1
      multiCluster:
        clusterName: platform-cluster
      network: platform-vpc

Above yaml file can be stored as istio-config.yaml(Note: This is not production ready, Only for demo purposes).

You can install istio in two ways:

4a. Direct installation based on the istio-config

istioctl install --context="${APP_CLUSTER}" -f istio-config.yaml
istioctl install --context="${PLATFORM_CLUSTER}" -f istio-config.yaml

4b. Generate manifests and apply it

istioctl manifest generate -f istio-config.yaml >> istio-installation-manifests.yaml

kubectl apply -f istio-installation-manifests.yaml --context="${APP_CLUSTER}"
kubectl apply -f istio-installation-manifests.yaml --context="${PLATFORM_CLUSTER}"

Step5: Check if EastWest Gateway is Installed in BothClusters

Above steps will create EastWest Gateway in both clusters. This will be deployed as Internal Network Load Balancers in AWS.

kubectl get svc istio-eastwestgateway -n istio-system --context="${APP_CLUSTER}"
kubectl get svc istio-eastwestgateway -n istio-system --context="${PLATFORM_CLUSTER}"

Since its internal LoadBalancer it won’t work from external world, You can validate it by trying out curl command from your local.

Step6: Create Cross Cluster Network Gateway(Both Clusters)

# expose-services.yaml
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: cross-cluster-network-gateway
  namespace: istio-system
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15443
        name: tls
        protocol: TLS
      tls:
        mode: AUTO_PASSTHROUGH
      hosts:
        - "*.local"

kubectl apply -f expose-services.yaml --context="${APP_CLUSTER}"
kubectl apply -f expose-services.yaml --context="${PLATFORM_CLUSTER}"

Without this communication between clusters through eastwestgateway is not accepted.

Step7: Create Remote secret(Both Clusters)

This step is mandatory, so that each cluster authenticate to other cluster’s API. In short words, it is the service discovery mechanism.

# This gets the kubeconfig of APP_CLUSTER and create it as secret in PLATFORM_CLUSTER
istioctl create-remote-secret \
    --context="${APP_CLUSTER}" \
    --name=app-cluster-01 | \
    kubectl apply -f - --context="${PLATFORM_CLUSTER}"

# This gets the kubeconfig of PLATFORM_CLUSTER and create it as secret in APP_CLUSTER
istioctl create-remote-secret \
    --context="${PLATFORM_CLUSTER}" \
    --name=platform-cluster | \
    kubectl apply -f - --context="${APP_CLUSTER}"

kubectl get secrets -n istio-system --context="${APP_CLUSTER}"
kubectl get secrets -n istio-system --context="${PLATFORM_CLUSTER}"

Istio MultiNetwork, MultiPrimary Setup is completed. Now we are going to create Federated Prometheus in the istio-service mesh.

Step 8: Federated Prometheus to get Logs from all the clusters centrally.

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/addons/prometheus.yaml --context="${APP_CLUSTER}"
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/addons/prometheus.yaml --context="${PLATFORM_CLUSTER}"

The above will create prometheus in both the clusters. Once done we need to update the configmap prometheus in the platform-cluster deployment so that it can get the logs from app-cluster federated endpoint. Update the ingress gateway endpoint of prometheus in the below commented section. Apply the below only in platform-cluster.

- job_name: 'federate-app-cluster-01'
      scrape_interval: 15s
      honor_labels: true
      metrics_path: '/federate'
      params:
        'match[]':
          - '{job="kubernetes-pods"}'
          - '{__name__=~"istio_(.*)"}'
          - '{__name__=~"pilot(.*)"}'
          - '{__name__=~"envoy(.*)"}'
      static_configs:
      - targets:
        - # NOTE: Replace with the actual ingress gateway endpoint where Prometheus is reachable.
        labels:
          cluster: app-cluster-01
    - job_name: 'federate-local'
      honor_labels: true
      metrics_path: '/federate'
      metric_relabel_configs:
      - replacement: 'platform-cluster'
        target_label: cluster
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ['istio-system']
      params:
        'match[]':
        - '{__name__=~"istio_(.*)"}'
        - '{__name__=~"pilot(.*)"}'

Istio completely removed and decommissioned the Mixer component in the Istio 1.8 release (November 2020). It was phased out in favor of the unified istiod control plane and native proxy extensions like WebAssembly (Wasm) and Envoy-based filters. Sometimes the sidecars won’t be emitting the istio-related metrics, in order to fix it, deploy telemetry in both clusters.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-telemetry-default
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus

What it does? It lets the istio that it should emit metrics in the prometheus provider format, so that you can see the istio related metrics.

Step 9: Deploy Centralized Kiali to Visualize the flow between clusters

Apply the below yaml only in the platform-cluster.

kubectl apply -f https://github.com/istio/istio/blob/master/samples/addons/kiali.yaml --context="${PLATFORM_CLUSTER}" 

After applying this if you dont see the multi cluster setup, it might be becuase of below line

      - name: "kiali-multi-cluster-secret"
        secret:
          secretName: "kiali-multi-cluster-secret"
          optional: true

You need to create kiali-multi-cluster-secret to make sure that kiali can access the workloads through kubernetes-api and can be visualized in the Kiali UI. Make sure you label the kiali-secret properly as well.

Get Syed Sulaiman A’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

You can use the same istioctl command to create secrets but rename the secret to ‘kiali-multi-cluster-secret’ [Note: Do not replace/delete the istioctl remote secret. Both are separate secrets and can be accessed by istio/kiali]. You can store the secret with multiple cluster secrets.

kubectl label secret kiali-multi-cluster-secret kiali.io/multiCluster=true \
  -n istio-system \
  --context="${PLATFORM_CLUSTER}"

Step 10: Deploy the application

I donot have an custom application built by me. So I used another user’s repository to check the deployment.

Application repository: https://github.com/sydsulai/aws-eks/tree/main/fleetman

Thanks to the developer https://github.com/dickchesterwood

I deployed 3 micro-services to the platform-cluster and another 3 to the app-cluster.

Wait for all pods to be in the running state.

kubectl get pods -n default --context="${PLATFORM_CLUSTER}"
kubectl get pods -n default --context="${APP_CLUSTER}"

Step 11:

Cross Cluster Service Entry(Platform to App Cluster)

# expose-services.yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: fleetman-service-entry
  namespace: default
spec:
  hosts:
  - fleetman-vehicle-telemetry.default.svc.cluster.local
  - fleetman-staff-service.default.svc.cluster.local
  - fleetman-webapp.default.svc.cluster.local
  location: MESH_INTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  endpoints:
  - address: <PLATFORM_EASTWESTGATEWAY_FQDN> # Update the EASTWESTGATEWAY_FQDN with the actual FQDN of the east-west gateway in the remote cluster
    ports:
      http: 15443
    labels:
      topology.istio.io/network: platform-vpc

Cross Cluster Service Entry(App to Platform Cluster)

# expose-services.yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: fleetman-service-entry
  namespace: default
spec:
  hosts:
  - fleetman-position-tracker.default.svc.cluster.local
  - fleetman-api-gateway.default.svc.cluster.local
  location: MESH_INTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  endpoints:
  - address: <APP_EASTWESTGATEWAY_FQDN> # Update the EASTWESTGATEWAY_FQDN with the actual FQDN of the east-west gateway in the remote cluster
    ports:
      http: 15443
    labels:
      topology.istio.io/network: app-vpc

Apply the above yaml using kubectl command.

For encryption apply the below yaml file in both clusters.

# expose-services.yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: cross-cluster-mtls
  namespace: default
spec:
  host: "*.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  exportTo:
  - "."

kubectl apply -f expose-services.yaml --context="${APP_CLUSTER}"
kubectl apply -f expose-services.yaml --context="${PLATFORM_CLUSTER}"

Whats happening? Sample Scenario below

Pod in platform-cluster calls fleetman-webapp.default.svc.cluster.local

↓ (Envoy sidecar intercepts)

↓ ServiceEntry says: “That service is at east-westgateway-1234.ap-south-1.elb.amazonaws.com:15443”

↓ DestinationRule says: “Use ISTIO_MUTUAL TLS for that connection”

↓ Sidecar establishes mTLS connection to app-cluster-1’s east-west gateway:15443

↓ Gateway (AUTO_PASSTHROUGH mode) forwards encrypted traffic as-is

↓ Platform-cluster’s east-west gateway receives it on :15443

↓ Destination pod’s sidecar decrypts and delivers to container

✓ End-to-end encrypted, authenticated cross-cluster call complete

Test the connectivity

1. Check the connectivity by exposing the fleetman-webapp service with Layer 7 Application Loadbalancer and try to navigate the UI.
2. Or you can deploy any test pods in the default namespace and curl the services by executing from that pod.

Debugging the issues:

I will write the issues I faced during the deployment in the next blog.

Stay tuned…!!!!

Reference Docs:

1. Istio Multi Cluster Documentation
2. Istio Certificate Trust Identity
3. Istio Deployment Models
4. AWS EKS Documentation
5. Monitoring MultiCluster Istio with Prometheus Federation
6. Hierarchial Federation
7. Prometheus Yaml
8. Istio FleetMan Repository