Backstage on Kubernetes with Istio Service Mesh

Introduction

Backstage is an extremely popular Internal Developer Platform (IDP) that many larger software development organizations are adopting to help reunite development teams that have diverged over time. Many organizations experience this drift away from standards and best practices and find themselves in a situation where teams are reinventing the wheel because they aren’t aware that a software component already exists that meets their current needs.

This is where Backstage shines. The platform serves as a catalog for software components, infrastructure resources, organizational entities, and software templates that can be used to onboard new projects quickly. Having all of these items in one, easy to use portal gives development teams the keys to success when starting a new project or even when they need a new component to an existing system or application.

For the purposes of this article, we’ll dive right into the technical details of running Backstage in a way that is scalable and secure. So, if you need more information about Backstage, check out the official details here.

The Problem

While Backstage is fully featured and comes with some really nice core functionality, it can be difficult to manage within a scaled deployment that allows for many users and lots of software templates being executed on the backend system. By default, the frontend and backend are deployed in a single process or container. This is nice for simplicity of running the application, especially on your local system, but this creates issues for scaling and performance that are often encountered in a production environment with many simultaneous users.

This article will cover how to break up the frontend UI and backend services to create a more scalable deployment using Kubernetes. We’ll also look at adding Backstage containers to a service mesh using Istio to help with service discovery, load balancing between pods, and automatic mTLS between pods with no need for DevOps teams to deploy certificates directly on each container in the pod.

Prerequisites

Here are some assumptions that will be made for the purpose of keeping this article on topic. To follow along, you should have:

• A clone of the Backstage code repository (located here) and a new Backstage base application (following these instructions)
• A running Kubernetes cluster with Docker Desktop, Minikube, or even a cloud-based cluster (EKS, AKS, GKE)
• A working knowledge of Kubernetes and how to deploy an application to a running cluster

Installing Istio Service Mesh

To support service discovery and help secure the Backstage containers, we’re going to deploy the Istio service mesh to the cluster that is hosting the Backstage IDP. Istio can support a wide range of complex configurations that align well with environments that require more secure deployments. This example will use a very simple Istio deployment and configuration that doesn’t take a detailed understanding of Istio, but still accomplishes the basic need for service discovery and mTLS pod-to-pod security within the Kubernetes cluster. Istio can be installed via the istioctl CLI, or via Helm. We’ll use the CLI for easier installation for this example.

For more information on installing Istio, see the official documentation here.

Download Istio using the follow shell command:

curl -L https://istio.io/downloadIstio | sh -

Change to the Istio folder (the directory may be named differently based on the version that is downloaded):

cd istio-1.30.1

Add the Istio CLI to your path:

export PATH=$PWD/bin:$PATH

Install Istio using the default install profile:

istioctl install

With the default profile, Istio will be installed with the istiod and istio-ingressgateway. We will need the ingress gateway to allow traffic into the Kubernetes cluster for Backstage.

NOTE: Istio is installed in Sidecar mode by default. You can use the ambient mode option, but it will require a different configuration than what is shown in this article.

Separating Backstage’s Frontend and Backend

By default, Backstage will run the frontend and backend of the application within the same process. To facilitate larger environments that have many active users, or environments that require more compute for the Backstage scaffolder, separating and frontend and backend for scalability makes more sense. To accomplish this, we need to adjust the Backstage code to separate the frontend, and create a Dockerfile for hosting the static frontend code using NGINX.

We need to start by editing the index.ts file in the packages/backend/src folder of the new Backstage application that we created using the Backstage CLI. Removing the import for the @backstage/plugin-app-backend package will prevent the backend service from loading the frontend app inside the backend process.

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

// Comment or remove the following line to remove the frontend
// backend.add(import('@backstage/plugin-app-backend'));
backend.add(import('@backstage/plugin-proxy-backend'));

Next, we need to create a new Dockerfile to support running the Backstage frontend on NGINX. We’ll create this file in the packages/app directory in the Backstage application folder. To help facilitate the build process, we’ll name this file frontend.Dockerfile. This will help when we build the Docker images inside a consolidated build script.

FROM nginx:mainline

RUN apt-get update && apt-get -y install jq && rm -rf /var/lib/apt/lists/*

COPY packages/app/dist /usr/share/nginx/html
COPY packages/app/docker/default.conf.template /etc/nginx/templates/default.conf.template

COPY packages/app/docker/inject-config.sh /docker-entrypoint.d/40-inject-config.sh

ENV PORT=80

We’ll also need a dockerignore file to prevent the Docker build from pulling in files that are not required by the frontend. To align with the Dockerfile name, this file should be named frontend.Dockerfile.dockerignore. Add the following in this new dockerignore file:

.git
.gitignore
.yarn/cache
.yarn/install-state.gz
dist-types
node_modules
.env
.env.local
.env.*
packages/backend
packages/*/src
packages/app/node_modules
plugins
*.local.yaml
!packages/app/dist

There are a couple of other files that need to be added to support this Dockerfile. In particular, there is a config file for NGINX and a script from Backstage to help with injecting the configuration into the NGINX container. We’ll create a docker directory under the packages/app directory for these files.

NOTE: If you are using an FQDN, you can change all references to localhost to be your FQDN.

The first file to create is named default.conf.template with the following contents:

server {
    listen       $PORT;
    # if using a FQDN, change the following line to match
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    root    /usr/share/nginx/html;

    include /etc/nginx/mime.types;

    location / {
        index  index.html index.htm;
        try_files $uri /index.html;
    }

    location /healthcheck {
        return 204;
    }

    location /api/ {
        proxy_pass http://backstage-backend:7007/api/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Next, create a file named inject-config.sh with the following contents:

#!/usr/bin/env bash

set -Eeuo pipefail

function inject_config() {
  # Read runtime config from env in the same way as the @backstage/config-loader package
  local config
  config="$(jq -n 'env |
    with_entries(select(.key | startswith("APP_CONFIG_")) | .key |= sub("APP_CONFIG_"; "")) |
    to_entries |
    reduce .[] as $item (
      {}; setpath($item.key | split("_"); $item.value | try fromjson catch $item.value)
    )')"

  >&2 echo "Runtime app config: $config"

  local main_js
  if ! main_js="$(grep -l __APP_INJECTED_RUNTIME_CONFIG__ /usr/share/nginx/html/static/*.js)"; then
    echo "Runtime config already written"
    return
  fi
  echo "Writing runtime config to ${main_js}"

  # escape ' and " twice, for both sed and json
  local config_escaped_1
  config_escaped_1="$(echo "$config" | jq -cM . | sed -e 's/[\\"'\'']/\\&/g')"
  # escape / and & for sed
  local config_escaped_2
  config_escaped_2="$(echo "$config_escaped_1" | sed -e 's/[\/&]/\\&/g')"

  # Replace __APP_INJECTED_RUNTIME_CONFIG__ in the main chunk with the runtime config
  sed -e "s/__APP_INJECTED_RUNTIME_CONFIG__/$config_escaped_2/" -i "$main_js"
}

inject_config

We’re also going to rename the Dockerfile in the packages/backend directory to help with the build. We’ll rename this file as backend.Dockerfile. The contents of this Dockerfile can remain untouched.

Get Adam Bauer’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

NOTE: If you want to use localhost for this deployment, you’ll need to update the following line in the backend.Dockerfile or the Backstage backend pod will throw security errors related to using localhost in a production environment:

# This is set to production by default
# Change NODE_ENV to development for localhost deployments
ENV NODE_ENV=development

We’ll also want to create a dockerignore file to prevent the backend container from pulling in files that are only required by the frontend application. Create a file named backend.Dockerfile.dockerignore in the packages/backend directory with the following contents:

.git
.gitignore
.yarn/cache
.yarn/install-state.gz
dist-types
node_modules
.env
.env.local
.env.*
packages/app
packages/*/src
packages/backend/node_modules
plugins
*.local.yaml

Updating the Backstage app-config.yaml

Now that the Dockerfiles are in place, and the frontend has been separated from the backend, we need to update the app-config.yaml file to tell Backstage where the parts of the application will be located. For this example, we’ll use localhost, but this can easily be any valid URL for where Backstage is running.

Since our Backstage app will be running on port 80, we’ll change the base URL of the app to remove port 3000 which is the default for new Backstage installations, as well as the backend.cors.origin so that CORS requests will be accepted. Edit the app-config.yaml in the root of your new Backstage application:

NOTE: If you are using an FQDN, you can change all references to localhost to be your FQDN.

The example below also adds configuration for the Postgres database that we’ll deploy in this example.

app:
  title: Backstage
  # Remove port 3000 from the baseUrl
  baseUrl: http://localhost

backend:
  # Remove port 7007 from the baseUrl
  baseUrl: http://localhost
  listen:
    port: 7007
  csp:
    connect-src: ["'self'", 'http:', 'https:', 'ws:', 'wss:']
  cors:
    # Remove port 3000 from the origin
    origin: http://localhost
    methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
    credentials: true
  database:
    client: pg
    connection:
      host: ${POSTGRES_HOST}
      port: ${POSTGRES_PORT}
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
  actions:
    pluginSources:
      - auth
      - catalog
      - scaffolder

The backend also has a baseUrl parameter, but this will remain unchanged as the backend service will still be running on the default port of 7007.

Building the Bundled Application

With the new Dockerfiles and application config in place, we’re ready to build the Backstage application with the separated frontend and backend containers.

Here is an example script that can be used to build the application code and Docker images. You’ll need to make sure the Docker engine is running and that you are in the root of the Backstage app you created before calling the build script.

#!/bin/bash
set -e

echo "Executing yarn install..."
yarn install

echo "Compiling TypeScript..."
yarn tsc

echo "Building Backstage backend..."

yarn build:backend

echo "Building Backstage frontend..."

yarn workspace app build

echo "Building backend Docker image..."

docker buildx build \
  -f packages/backend/backend.Dockerfile \
  -t backstage-backend:latest \
     .

echo "Building frontend Docker image..."

docker buildx build \
  -f packages/app/frontend.Dockerfile \
  -t backstage-frontend:latest \
     .

Preparing the Kubernetes Manifests

We’re now ready to set up the Kubernetes deployment for the Backstage containers. To make this a bit easier, we’ll be using the Kustomize feature to help with the deployment rather than create a bulky manifest file.

Start by creating a kubernetes directory outside of your Backstage application directory, and then add the following files inside the kubernetes directory. Having this located outside of your Backstage project will prevent the files from being pulled into any Docker images. You could also exclude them in your dockerignore files, if you prefer.

backstage-namespace.yaml:
This will create the namespace in your cluster and configure it to auto-inject any deployment into the Istio service mesh.

apiVersion: v1
kind: Namespace
metadata:
  name: backstage
  labels:
    istio-injection: enabled

backstage-backed-deployment.yaml:
This will set up the backend service deployment and the Kubernetes service that will expose the pods to the cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backstage-backend
  namespace: backstage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backstage
  template:
    metadata:
      labels:
        app: backstage
        component: backend
    spec:
      containers:
        - name: backstage-backend
          image: backstage-backend:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 7007
          readinessProbe:
            httpGet:
              port: 7007
              path: /.backstage/health/v1/readiness
          livenessProbe:
            httpGet:
              port: 7007
              path: /.backstage/health/v1/liveness
          # These values should be secrets in a production environment
          env:
            - name: POSTGRES_HOST
              value: postgres
            - name: POSTGRES_PORT
              value: "5432"
            - name: POSTGRES_USER
              value: backstage
            - name: POSTGRES_PASSWORD
              value: backstagepassword
---
apiVersion: v1
kind: Service
metadata:
  name: backstage-backend
  namespace: backstage
spec:
  selector:
    app: backstage
    component: backend
  ports:
    - port: 7007
      targetPort: 7007
      protocol: TCP

backstage-frontend-deployment.yaml:
This will set frontend NGINX deployment, as well as the related resources for ingress into the cluster. NOTE: If you are using an FQDN, you can change all references to localhost to be your FQDN.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backstage-frontend
  namespace: backstage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backstage
  template:
    metadata:
      labels:
        app: backstage
        component: frontend
    spec:
      containers:
        - name: backstage-frontend
          image: backstage-frontend:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 80
          # These values should be secrets in a production environment
          env:
            - name: POSTGRES_HOST
              value: postgres
            - name: POSTGRES_PORT
              value: "5432"
            - name: POSTGRES_USER
              value: backstage
            - name: POSTGRES_PASSWORD
              value: backstagepassword
---
apiVersion: v1
kind: Service
metadata:
  name: backstage-frontend
  namespace: backstage
spec:
  selector:
    app: backstage
    component: frontend
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: backstage-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      # Set hosts to wildcard for localhost or FQDN if available
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backstage-frontend
  namespace: backstage
spec:
  # Set hosts to wildcard for localhost or FQDN if available
  hosts:
    - "*"
  gateways:
    - istio-system/backstage-gateway
  http:
    - route:
        - destination:
            # This host must be the cluster FQDN
            host: backstage-frontend.backstage.svc.cluster.local
            port:
              number: 80
            subset: v1
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backstage-frontend
  namespace: backstage
spec:
  # This host must be the cluster FQDN
  host: backstage-frontend.backstage.svc.cluster.local
  subsets:
    - name: v1
      labels:
        app: backstage
        component: frontend

postgres-deployment.yaml:
This will set up the PostgreSQL database in your cluster using a stateful set. You’ll need to make sure that your app-config.yaml file is properly configured to connect to the PostgreSQL instance. You can also use the SQL Lite database, but this method is preferred so that state can be maintained over time.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: backstage
spec:
  replicas: 1
  serviceName: postgres
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      name: postgres
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:15.5-alpine
          imagePullPolicy: 'IfNotPresent'
          ports:
            - containerPort: 5432
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgresdb
              subPath: data
          # These values should be secrets in a production environment
          env:
            - name: POSTGRES_HOST
              value: postgres
            - name: POSTGRES_PORT
              value: "5432"
            - name: POSTGRES_USER
              value: backstage
            - name: POSTGRES_PASSWORD
              value: backstagepassword
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: postgresdb
        namespace: backstage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 8G
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: backstage
spec:
  type: ClusterIP
  selector:
    app: postgres
  ports:
    - port: 5432

kustomization.yaml:
This file will serve as the master file for the deployment.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - backstage-namespace.yaml
  - backstage-frontend-deployment.yaml
  - backstage-backend-deployment.yaml
  - postgres-deployment.yaml

Deploying Backstage to Kubernetes

We’re now ready to deploy the Backstage application to a Kubernetes cluster. Since the namespace is defined with Istio injection, the deployed pods will be automatically meshed when they are deployed, and the Istio ingress gateway will allow access to Backstage.

To deploy the Kustomization manifest, switch to the kubernetes directory where the YAML files are saved, and run the following command to deploy the Backstage resources to the cluster:

kubectl apply -k .

Validating the Deployment

NOTE: Once the deployment is completed, you may see some failures in the backend container. These usually happen when the postgres pod has not fully initialized the database immediately after the first deployment. Simply restart the backstage-backend pod to fix these errors.

Once all of the pods are running, you can open your browser and navigate to http://localhost (or the FQDN that was configured for your instance). You’ll now see the Backstage UI with the example catalog created by the base application setup.

Backstage on Kubernetes with Istio Service Mesh

Next Steps

Using this example, you can quickly move into a production deployment with a configuration that matches your target environment. There are a number of options for creating additional backend pods where you can isolate specific Backstage services, like the scaffolder, into pods dedicated for those services. This serves to improve application performance and prevent backend services from affecting the frontend process.

Scaling is also made easier by simply increasing the replica count for the frontend or backend in your deployment to help with additional user load. Istio also provides easy ways to use TLS certificates within your cluster so that all ingress traffic for your cluster is secure.

With this configuration in place, your Backstage instance will be ready to scale and perform well in larger environments with high user load.