By Stephan Schneider — Expert Associate Partner, Navin Agarwal — Principal Cloud Engineer, Thomas Delaet — Partner, and Remy Paternoster — Partner, McKinsey Digital

How can tech leaders unleash the full potential of their software development talent? By empowering their developers, creating an environment ripe for innovation, and improving their productivity with the right tools and processes. These factors contribute to what we call “developer velocity” — a key driver of business success. Previous McKinsey research has found that companies with top-quartile developer velocity outperform bottom-quartile companies by four to five times (measured in revenue CAGR).

Reducing the cognitive and operational load on developers is a critical step toward increasing product innovation and velocity. The idea of “you build it, you run it” has empowered software development teams to take ownership over their products and release them more frequently and with higher quality. However, it has also placed tremendous strain on software developers. They need to stay current on the latest software-engineering practices and tooling, maintain in-depth knowledge of running and integrating a large breadth of cloud service offerings, and deliver on the ever-increasing velocity of product output that many organizations expect.

A modern developer platform can go a long way toward boosting developer velocity and elevating the developer experience. By acting as a self-serve “one-stop shop” for developers, such a platform reduces manual work, increases process efficiency, and propels standardization, security, and compliance by design. In other words, it allows developers to focus on what they do best — writing code to solve a customer problem.

While every platform is different, certain best practices have emerged across companies and industries. Drawing from our experience, we have developed a set of design principles and a reference architecture that can serve as a useful blueprint for a best-in-class developer platform and enable DevOps at scale.

Eight design principles for developer platforms

Platform-engineering teams should follow a set of core design principles when building the developer platform:

1. Focus on the user. Because developers are the most important “customers” of a developer platform, they need to be heavily involved in a platform’s design, prioritization of features, and testing to make sure it is fit for purpose, user-friendly, and fully self-service. A user-centric design that addresses the needs of developers can make all the difference and drive adoption.
2. Run the platform team like a start-up. IT leaders should establish a small, central team that not only owns the platform but is also responsible for marketing it (and has the resources to do so). The team also needs sufficient capacity to ensure the successful implementation of the platform through community forums, technical support, and clear and concise documentation.
3. Decentralize the contribution model. While ownership falls to a central team, the contribution model should allow for decentralized, continuous development. As in an open-source model, contributors — for example, application development (app dev) teams — should be able to share plug-ins and propose changes, but the platform team should retain the final say over what is approved and accepted in the platform. It is important to acknowledge that app dev teams across the organization are at different levels of maturity, capability, and engagement — meaning not all are ready or willing to contribute to the platform from day one. To establish an inner-sourcing culture, organizations will need to support an operating model in which teams with the right skills and knowledge can contribute. They also need to maintain an upskilling mechanism for teams to grow over time into contributing roles.
4. Build golden paths, with some cages. Developers should be free to choose the abstraction level that suits their needs. While a developer platform should provide a set of predefined patterns or “golden paths” for developers to follow — in addition to some “cages” they are not allowed to leave — it also needs to provide flexibility for developers to implement more-complex use cases or edge cases.
5. Drive standardization by design. Enabling self-service requires platform engineers to establish secure foundations and define preconfigured application patterns to ensure every resource created through the platform is secure, compliant, and well architected. However, application patterns only standardize the first deployment, after which teams can edit and drift away from the standard. To ensure resources stay compliant and in line with the architected application patterns, the platform needs a mechanism that enforces these standards with every single deployment — for instance, via policies on the cloud-provider level, via automated checks in the pipeline or source code, or via centralized components such as a platform orchestrator that enforces standardization by continuously generating apps and infrastructure against the pattern during deployment.
6. Get the most from existing capabilities. Every organization has at least some individual components of a developer platform in place (such as base and isolation zones in the cloud), and standard components such as backlog management, continuous integration and continuous deployment (CI/CD) toolchain, and test suites are widely available in the market. A modern developer platform serves as the glue between these individual components and offers an opinionated set of design choices on how to use and configure them (baked as code into the platform) to fulfill developer needs.
7. Let developers choose their platform interface. The developer platform should not break developer workflows or force developers to use any specific interface. A code-based workflow should be the default, with options to use a user interface (UI), command line interface (CLI), or application programming interface (API).
8. Define success as usage. We find that the true marker of success is an organization’s ability to drive traffic to the platform and entice developers to use it in their day-to-day work. A platform that enhances the experience of developers and addresses their pain points can create pull from the organization. Ultimately, developers should use the platform because they want to, not because they have to. This can be achieved via gamification and rewards (for example, by rewarding app dev teams and individuals for faster adoption) or enforcing certain standards (which are out-of-the-box ready in the platform).

Developer platform reference architecture

Modern developer platforms require interchangeable components and provide flexibility to suit both simple and complex (multicloud or hybrid) architectures. The exhibit below illustrates a commonly deployed AWS architecture that can provide a starting point for a modern developer platform. However, IT leaders should begin by assessing their existing architectures and developer pain points, and then incorporate the components that they already have in place. The components and tools referenced below exemplify an AWS-based setup. All components are interchangeable, and similar reference architectures can be implemented for GCP, Azure, OpenShift, or any hybrid setup. Individual components can be replaced with any homegrown, on-premises, or other cloud-native solutions.

The developer platform reference architecture includes five main components:

• The developer control plane provides the interfaces through which developers can interact with the platform (IDE, API, CLI, GUI, GitOps, and so on). It is where developers can access the structured catalog with the components and documentation they need, and where they can implement, review, and deliver changes.
• The integration and delivery plane has the tools that build, store, configure, and deploy all requests coming from the developer control plane — including CI/CD, registries, and platform orchestration.
• The monitoring and logging plane provides real-time metrics and logs for applications and infrastructure, allowing developers to observe, monitor, and make data-driven decisions.
• The security plane manages confidential information and identities to protect sensitive data — for example, by storing, managing, and securely retrieving API keys and passwords.
• The resource plane includes all the components necessary to run applications delivered through the platform (as predefined in application patterns and workload specifications).

To assemble these planes into an end-to-end architecture, the platform engineering team must wire the individual components of the planes to one another, wire one plane to the other, and test the raw end-to-end flows to ensure smooth interplay.

To shortcut this process, we have defined this reference architecture for multicloud setups as code (AWS, Azure, GCP, and OpenShift), which enables an organization to set up a basic developer platform within a few hours. The platform setups are codified as Terraform scripts, and the workload specifications for common application patterns (such as three-tier application, microservice, and message bus) are implemented both as Terraform scripts and by using the open-source workload specification Score. Later, a platform orchestrator (a rule engine that matches resource requests to predefined default configurations provided by the platform team) dynamically creates the final configurations on each deployment. Multiple options can come into play as orchestrator, including cloud-native (such as AWS CloudFormation), homegrown (such as using Terraform and Argo CD), or off-the-shelf (such as Humanitec) services.

Contribution and governance model

Because developer platforms are designed to enable standardization, security, and compliance, it is critical to clearly define the setup and governance of the repositories from the beginning. We differentiate between repositories owned by development teams and those owned by platform teams: the former is based on workload, and the latter is shared across the organization.

On the development team level, the workload-based repository should include the source code, workload specification, docker file, and pipeline YAML. The “traditional” app configurations and infrastructure-as-code (IaC) files are replaced with a workload specification. This allows developers to describe their architecture (the relationship to other workloads and dependent resources) in a general, abstract, and environment-agnostic way. For instance, a workload specification could read, “I am a workload of the name ‘Python service.’ I have a dependency on a database of type PostgreSQL, an AWS S3 bucket, and a DNS.”

Meanwhile, the platform team should be responsible for developing the following cross-cutting elements:

• Resource drivers and infrastructure as code: A catalog of all resources made available by the platform team (such as cloud-native components and on-premises) and templates for dynamically created resources
• Resource definitions: Guidance on which resource drivers to use in a specific context; for example, “use this type of AWS S3 bucket” if the context of the deployment is a “staging” environment; use another one if it is “production”
• Workload profiles: Default configuration details for application configurations, such as CPU minimum allocation, labels and annotations, and sidecar containers
• Automation or compliance: Governance defined as code in the platform — for example, definitions of environment progression, deployment automations, and sign-off rules

The objective of this governance model is to provide development teams with ownership over their code and freedom to deliver it to production — as long as it stays within the guardrails defined by the platform team.

The goal of this approach is to cover 80 percent of the organization’s requirements with a few app patterns and resources (centrally defined by the platform team) and enable the rest to be defined via a decentralized contribution model (such as an open-source model), with the platform engineering team retaining final approval over contributions.

Say a developer needs an ArangoDB database (and has a valid business reason to stray off the default components defined in the platform), but the platform doesn’t support the definition yet. Assuming the platform team grants the rights, the developer should then expand the resource definitions — effectively “teaching” the platform how and when to deploy an ArangoDB — and send a pull request to the organization-wide repository that the platform team can then review and approve. If the next user gets into a similar situation and the platform team has globally accepted the definition, the procedure is already fully automated.

Why is this approach “better” than the current CI/CD flow? Because it allows continuous standardization of the existing estate through the platform. Imagine updating labels and annotations across all your workloads or updating your PostgreSQL definition from version 14 to version 15. Or say you want to reuse infrastructure definitions without having to go through security review every time. This platform design makes doing so possible while enabling platform teams to drive standardization, security, and compliance by design. With this design, platform teams are also able to optimize the load and infrastructure configuration organization-wide (instead of by team) and drive the automation of FinOps policies (FinOps as code) to reduce cloud spend systematically.

***

Developer platforms are a growing trend, but few organizations have fully cracked the code. By starting with a set of design principles and a reference architecture based on best practices and enterprise learnings, IT leaders can more confidently take the first steps toward building a modern developer platform and unlocking the full potential of their software development teams.

Platform as code: Reference architectures to simplify developer platforms was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Maksood Mohiuddin — Senior Cloud Engineer II, Alex Dequevedo — Senior DevOps Engineer I, Thomas Neff — Senior DevOps Engineer I, and Marco Marulli — Principal Lead II

Creating a modern, three-tier serverless application on a cloud service provider (CSP) given a containerized application is a common task for many cloud engineers. The challenge lies in being able to efficiently leverage the CSP’s breadth of services for different application tiers while expediting development efforts and reducing costs and complexity. In this article, we provide a step-by-step guide for deploying a modern, three-tier serverless application on Google Cloud that accelerates deployment time while reducing cost and complexity, along with detailed explanations of the services used in this implementation.

Architecture overview

We will use an example application titled “Amazing Employees,” in which we employ a three-tier microservices architecture that comprises the frontend (the presentation tier), backend (application tier), and a Firestore database (data tier). The frontend, which is developed using Angular, interacts with users, while the backend, which is built with Python Flask, manages business logic. “Amazing Employees” consists of a single frontend and backend container. However, microservices architectures are scalable: adding more containers can facilitate more features. Infrastructure for the application is defined through Terraform, and Cloud Build manages the application’s building and deployment.

Why serverless?

Serverless technology offloads much of the complex work of managing infrastructure to CSPs, enabling developers to concentrate on creating and deploying application code. Serverless offerings provide many benefits, including the following:

• nearly infinite automatic scaling
• high availability
• simplified infrastructure management
• potential cost savings (pay only for what you use)
• faster development cycles

Serverless technologies help enable several highly scalable architecture types, such as a microservices architecture. Microservices break down applications into smaller, independently deployable units. With the ability to scale individual microservices dynamically and isolate faults, developers can create highly responsive and cost-effective solutions. While serverless technology is not required to create a microservices application, it greatly simplifies the management and scaling aspects, making it an ideal choice for organizations looking to optimize resource utilization and streamline development efforts.

Another common architecture for serverless applications is an event-driven architecture, in which applications respond dynamically to events triggered by various sources, such as user actions, data changes, or external system interactions. It allows developers to design systems that automatically scale in response to incoming events, ensuring optimal performance during peak loads and cost savings during quieter periods.

Application services

For the frontend and backend of our “Amazing Employees” application, we leverage Cloud Run, a fully managed, serverless container environment from Google Cloud. Cloud Run is based on Knative, a serverless container technology that Google Cloud pioneered. Cloud Run is often easier to use than other container orchestration services, such as Google Kubernetes Engine (GKE), because it abstracts the infrastructure layer away and there is no need to manage the underlying Kubernetes clusters. Cloud Run automatically scales containers based on incoming traffic. The latest version of Cloud Functions — another popular choice to build serverless frontends or backends in Google Cloud — uses Cloud Run behind the scenes.

Of course, what you gain in simplicity, you lose in control. This is true for all managed services in the cloud. If your use case requires you have full control over your containers (such as configuring networking, scaling, and resource allocation at a granular level), you may need to use GKE instead of Cloud Run. Whenever possible, we recommend using fully managed services such as Cloud Run because, simply put, they make your life much easier.

API Gateway is used to secure traffic between frontend and backend Cloud Run instances. Generally speaking, using API Gateway is a best practice when building a microservice architecture because it improves security, performance, and flexibility while reducing complexity. Google Cloud’s API Gateway has three main components: the API, the gateway, and the config. The config uses OpenAPI specs and can be viewed in the infra directory. Note that two variables are used, which allows us to deploy the config without making manual changes. The first variable is ${project_id}, which is provided in variables.tf, and the second is ${url}. The ${url} variable value is provided by a Terraform data block that checks the URL of the backend Cloud Run instance and adds it to the config. In both cases, variable substitution is done through Terraform and can be viewed in infra/api-gateway.tf.

For the application database, Firestore was a natural choice due to its seamless integration into our serverless architecture. Firestore, a NoSQL database from Google Cloud’s Firebase service, was selected primarily for its ability to offer automatic scaling, robust querying support, advanced security rules, and dependable application authentication. In the context of our application, we use Firebase Authentication for user logins and employ basic CRUD database operations for managing employee data.

Infrastructure and deployment

For infrastructure as code, we use Terraform, an open-source tool that HashiCorp developed. Terraform allows you to define, manage, and provision infrastructure resources across various cloud providers, including Google Cloud, in a declarative manner.

Optionally, you can use Cloud Storage to store Terraform state. Cloud Storage is an object storage service within Google Cloud that can store unstructured data such as files and media and offers scalability to manage vast amounts of data across multiple storage classes. Cloud Storage offers data durability through replication and provides accessibility with low latency; security features such as access control and encryption; and versioning, life cycle management, and integration with other Google Cloud services.

To build the frontend and backend for the application, we use Cloud Build, a serverless continuous integration (CI) and continuous delivery (CD) platform from Google Cloud that allows you to execute builds in Google Cloud.

To store the container image, we leverage Artifact Registry, a managed service from Google Cloud that provides secure and scalable repositories for software artifacts such as packages for Java, Node, or Python, but in this case, we will use it to store Docker images.

Below is a description of the steps and a high-level diagram showing the flow of the Cloud Build job:

1. User triggers Cloud Build from local repository using Google Cloud commands.
2. Docker image is built from the current working directory.
3. Docker image is pushed to Artifact Registry.
4. Cloud Run instances are deployed from the Docker image.

A note about cost

This article is tailored to a cloud engineer who likes to explore serverless offerings from Google Cloud. As such, a typical deployment of the application for noncommercial use should be covered under Google Cloud Free Program. You can visit the Google Cloud free-tier products page to learn more: https://cloud.google.com/free. Following are the services and the related cost (as of September 2023) we will use for this tutorial:

• Cloud Run free tier offers 2 million requests monthly, 360,000 GB-seconds of memory, 180,000 vCPU-seconds of compute time, and 1 GB network egress (North America only).
• Firestore free tier offers 1 GB storage per project and 50,000 reads, 20,000 writes, and 20,000 deletes per day, per project.
• API Gateway is not part of the free-tier offering, but it offers up to 2 million API calls per billing account per month for free.
• Cloud Build free tier offers 120 build minutes per day.
• Artifact Registry free tier offers 0.5 GB storage per month.

Here, we will explore one example of a cost estimation for our serverless three-tier web application with example numbers. Actual costs will vary depending on usage.

We will use the following assumptions for our cost estimation:

• Cloud Run is used as a service, with CPU allocated only when running.
• Cloud Run instances have 1vCPU and 512 MB memory.
• There are 10,000 monthly users.
• Each user averages 400 requests per month.
• Each request takes 1,000 milliseconds to complete.
• 60 percent of requests are database reads.
• 40 percent of the requests are database writes.
• Firestore has less than 15 GB of data.
• All resources are deployed in the same region.

API Gateway:

2 million to 1 billion requests per month = $3.00

Firestore:

15 GB storage = $2.52 per month

Document reads:

50,000

$0.036 per 100,000 documents

Document writes:

20,000

$0.108 per 100,000 documents

100 users * 40,000 requests per month = 4 million requests per month ≈ 2.4 million reads and 1.6 million writes per month

Read cost: 2.4 million reads / 30 ≈ 80,000 requests per day — 50,000 free requests = 30,000 paid requests per day = $0.036 per day for read requests * 30 = $1.08 per month

Write cost: 1.6 million writes / 30 ≈ 54,000 requests per day — 20,000 free requests = 34,000 paid requests per day = $0.216 per day for write requests * 30 = $6.48 per month

Egress:

Within same region = $0.00

Cloud Run:

CPU allocation time: 200,000 vCPU-second = $0.48

Memory allocation time: 100,000 GiB-second = $0.00

Number of requests: 4,000,000 = $0.80

Cloud Run total: $1.28 * 2 instances = $2.56 per month

Monthly costs:

API Gateway: $3.00

Firestore: $10.08

Egress: $0.00

Cloud Run: $2.56

Total = $15.64 per month

Step-by-step guide

Setting up the Google Cloud project

Projects are required to manage and organize resources in Google Cloud. We recommend creating a new project for this tutorial. This tutorial assumes that you have a Google Cloud project with admin (owner) access and you have access to Google Cloud Shell with either knowledge of command-line editors such as nano or vim or access to the Cloud Shell Editor. Cloud Shell Editor is a text editor built right into Cloud Shell and is a great option if you are not familiar with command-line editors.

To access your project in the Google Cloud console, navigate to the Google Cloud console, log in (or create an account), and select your project. If you do not already have a Google Cloud project, navigate to Menu > IAM & Admin > Create a Project and follow the prompts in the console.

Initial setup

For simplicity, we use a single repository to hold all of the application, infrastructure, and pipeline code. For the initial setup, open a Cloud Shell session in the Google Cloud console for the project. Cloud Shell represents an internet-based platform for development and operational tasks and can be conveniently reached through a web browser. Google Cloud offers this service at no cost. Within this platform, you can oversee your assets using an online terminal that is equipped with various tools. For an in-depth dive of the preinstalled tools, please see https://cloud.google.com/shell/docs/how-cloud-shell-works#tools.

You can open a new Cloud Shell session by visiting https://console.cloud.google.com/?cloudshell=true.

If not already updated, update Google Cloud Shell console to use your project:

gcloud config set project [YOUR GOOGLE CLOUD PROJECT]

gcloud config list

Step 1: Clone the GitHub repository

In the Cloud Shell session, git clone the reference application. You can clone this repository to start. However, we recommend you fork the repository and clone the forked repository. As part of this step-by-step guide, we will provide instructions on how to change the code to deploy the three-tier serverless application in your Google project.

git clone [original https://github.com/maksoodmohiuddin/google-cloud-serverless-app-pattern.git or forked repo] cd google-cloud-serverless-app-pattern

From here you can select “Open Editor” to launch the Cloud Shell editor.

Step 2: Update Google project ID and region

Next, let’s update project_id and location with the Google Cloud project you are using and a target Google Cloud region (such as “us-west2”) where you would like to deploy this application. Our example shows nano as an editor. However, you can choose to use another available editor that you prefer.

cd infra
nano variables.tf

variable "project_id" {
  description = "Project ID of the GCP project where resources will be deployed"
  type        = string
  default     = "PLEASE UPDATE WITH YOUR GOOGLE PROJECT ID"
}

variable "location" {
  description = "Location (region) where resources will be deployed"
  type        = string
  default     = "PLEASE UPDATE YOUR GOOGLE CLOUD REGION"
}

Please choose a region that supports Cloud Build, Cloud Run, Firebase, API Gateway, and Artifact Registry. For personal accounts, Cloud Build is restricted to certain regions. Therefore, if you are using a personal account, we recommend using one of the regions below:

• us-west2
• asia-east1
• australia-southeast1

To learn more about Google Cloud regions and supported regions for the services for this app, please visit the following:

• General region picker: https://cloud.withgoogle.com/region-picker/
• Regions that support Firestore: https://cloud.google.com/firestore/docs/locations
• Regions that support Cloud Build: https://cloud.google.com/build/docs/locations
• Regions that support Cloud Run: https://cloud.google.com/run/docs/locations
• Regions that support Artifact Registry: https://cloud.google.com/artifact-registry/docs/repositories/repo-locations
• Regions that support API Gateway: https://cloud.google.com/api-gateway/docs/deployment-model

Deploying initial infrastructure

In this section, we will deploy the initial infrastructure with Terraform. Due to dependency mapping that is not known to Terraform, we will need to target specific resources and set up the application sequentially.

By default, Terraform stores state locally in a file named terraform.tfstate. For purposes of this tutorial, you can use the local state. However, if you prefer, state can be stored in remote state using Google Cloud Storage. Remote state is always preferred for projects with multiple developers. If you are especially savvy with Terraform, you may find it easier to create the bucket with Terraform local state, and then migrate the state into the bucket. See the Google Cloud documentation for more details.

Step 1: Initialize Terraform

terraform init

Step 2: Validate Terraform plan

terraform plan

If prompted to authorize Cloud Shell, please select “Accept” to continue. This is because Cloud Shell needs permission to use your credentials for the Google Cloud command, and clicking “Authorize” grants permission to this and future calls.

The output should be the following:

Plan: 18 to add, 0 to change, 0 to destroy

Step 3: Apply Terraform for the initial resources

terraform apply

When prompted, type “yes” to deploy the resources. It should match the plan.

Deploying backend, API Gateway, and frontend

Step 1: Update the backend Python Flask application code to use your Google project

Navigate to the backend folder and open the firestore.py file:

cd ../app/backend/ nano -l firestore.py

Update line 10 client with the Google Cloud project you are using:

client = firestore.Client(project="PLEASE_UPDATE_PROJECT_ID")

Step 2: Trigger Cloud Build to deploy backend

First, you can review the Cloud Build file and use the Google Cloud command line to deploy the backend Cloud Run based on the Cloud Build file:

cat backend-cloudbuild.yaml gcloud builds submit --region=[YOUR GOOGLE CLOUD REGION] --config backend-cloudbuild.yaml e.g. gcloud builds submit --region=us-west2 --config backend-cloudbuild.yaml

Note that the Cloud Build run can take several minutes to finish the run. This is expected. This Cloud Build file builds a Docker image for the backend service, pushes it to Artifact Registry, and then deploys the images from the Artifact Registry on a Cloud Run instance.

Step 3: Review the Swagger specification

We use API Gateway with OpenAPI Swagger specifications to connect the backend Cloud Run to API Gateway and set up the API Gateway configuration. Review the api-gateway--espv2-definition.yml.tmpl file under the infra folder.

First, use cd to go back to the infra directory and enable the enable_api_gateway flag to true:

cd ../../infra cat api-gateway--espv2-definition.yml.tmpl

You can learn more about OpenAPI specifications for API Gateway, Swagger, and Firebase as authentication for API Gateway by visiting the following:

https://cloud.google.com/api-gateway/docs/openapi-overview

https://swagger.io

https://cloud.google.com/api-gateway/docs/authenticating-users-firebase

Step 4: Deploy API Gateway

While still in the infra directory, enable the enable_api_gateway flag to true:

nano variables.tf

variable "enable_api_gateway" { description = "Feature flag to enable/disable API Gateway. Leverage this to deploy infra sequentially." type = bool default = true }

Run the Terraform plan and then apply:

terraform plan

terraform apply

When prompted, type “yes” to deploy the resources. It should deploy three resources. Note that API Gateway typically takes longer to deploy than other resources. This is normal.

Step 5: Configure Firestore database via Firebase console

Navigate to https://console.firebase.google.com/ and Create/Add Project > Select your Google Cloud project from the dropdown > Accept Terms > Continue > Confirm Plan > Choose to enable Google Analytics or not > Continue.

In the landing page of your Firebase project, add your app as a web app to Firebase.

Alternatively, under Project Overview, click on the gear icon, navigate to Project Settings, and register your app as a web app to Firebase.

Register your app with your chosen name (for example, you can use your Google project name).

Once your app is registered, under the SDK setup and configuration section (you can toggle to the configuration section), copy the value for const firebaseConfig and paste the value into app/frontend/src/environments/environments.ts.

Warning: Make sure to not check in the values of environments.ts into a git repository because this is sensitive information.

cd ../app/frontend/src/environments/ nano -l environment.ts

export const environment = { firebase: { apiKey: "PLEASE UPDATE", authDomain: "PLEASE UPDATE", projectId: "PLEASE UPDATE", storageBucket: "PLEASE UPDATE", messagingSenderId: "PLEASE UPDATE", appId: "PLEASE UPDATE", measurementId: "PLEASE UPDATE" // if measurement is enabled }, production: false };

While still in Firebase console, enable Google as an authentication provider for Firebase with the following steps: All Products > Authentication > Get Started > Google > Enable > Add Support Email > Save.

Step 6: Set the frontend API Gateway configuration

First, run the following command to get the api-gateway configuration:

gcloud api-gateway gateways describe employee-gateway --location LOCATION --project PROJECT_ID --format 'value(defaultHostname)'

The output will look like the following: employee-gateway-#.#.gateway.dev

Then update the three URLs (lines 13–15) in app/frontend/src/employee/services/firestore.service.ts; keep the path (for example, employee) as is.

cd ../employee/services

nano -l firestore.service.ts

addEmployeeUrl = 'https://employee-gateway-#.#.gateway.dev/employee'; employeesUrl = 'https://employee-gateway-#.#.gateway.dev/employees'; deleteEmployeeUrl = 'https://employee-gateway-#.ue.gateway.dev/employee';

Step 7: Trigger Cloud Build to deploy the frontend

Navigate to the frontend folder, review the Cloud Build file, and use the Google Cloud CLI to deploy the backend cloud instance.

cd ../../.. cat frontend-cloudbuild.yaml gcloud builds submit --region=[YOUR GOOGLE CLOUD REGION]] --config frontend-cloudbuild.yaml e.g. gcloud builds submit --region=us-west2 --config frontend-cloudbuild.yaml

Step 8: Update Firebase authorized domain with frontend Cloud Run URL

The frontend Cloud Run URL needs to be authorized for OAuth operations in the Firebase app.

You can use Google Cloud to get the Cloud Run URL:

gcloud run services describe SERVICE --region REGION --format 'value(status.url)'

e.g. gcloud run services describe amazing-employees-frontend-service --region us-west2 --format 'value(status.url)'

The output should be the following: https://amazing-employees-frontend-service-###.a.run.app

Copy that URL and add to the OAuth redirect domains list in the Firebase console.

Browse to https://console.firebase.google.com/ and select your project.

Click on Authentication > Settings Tab > Authorized Domains > Add Domain, and paste your frontend Cloud Run URL.

Final step: Validate end-to-end app

Browse to your frontend Cloud Run URL (https://amazing-employees-frontend-service-###.a.run.app), and use a Google account to log in to the app — your Google Cloud serverless application pattern using Cloud Run, API Gateway, and Firebase is up and running.

Clean up

We will not be able to run a Terraform destroy here due to API Gateway configuration dependencies.

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete the project.

Deleting a project has the following consequences:

• If you used an existing project, you’ll also delete any other work you’ve done in the project.
• You can’t reuse the project ID of a deleted project. If you created a custom project ID that you plan to use in the future, delete the resources inside the project instead. This ensures that URLs that use the project ID, such as an appspot.com URL, remain available.

To delete a project, do the following:

• In the Cloud console, go to the projects page (https://console.cloud.google.com/iam-admin/projects).
• In the project list, select the project you want to delete and click “Delete.”
• In the dialog, type the project ID and click “Shut down” to delete the project.

Summary

This article delved into the deployment of a serverless three-tier application on Google Cloud. We explored how to use Cloud Run for both an Angular containerized frontend and a Python Flask containerized backend while leveraging Firestore for our database needs. Terraform and Cloud Build were highlighted for efficient infrastructure setup and deployments. For those eager to be hands-on, we provided a detailed step-by-step guide to ensure you can replicate and understand each phase of the deployment.

By understanding these key components and technologies, readers should be better poised to tackle serverless projects on Google Cloud.

Building a serverless application on Google Cloud to accelerate deployment time with reduced cost… was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Aditya Saxena — Expert Associate Partner, Deepak Joshi — Principal Product Manager, and Pallavi Misra — Product Specialist II, Build by McKinsey

Congratulations! The management team is impressed with your product vision and idea, and it has approved the budget to execute the idea. The idea is among the top three priorities for your organization this quarter, and the plan is to launch it in the market by next quarter. This is terrific and much-awaited news! Now starts your journey to convert your idea into reality. But here is a glaring fact that stares at you. According to research published in 2021¹, there are over 30,000 new products introduced every year (physical and digital), and 95% fail, telling us that converting product ideas into reality is no cakewalk. The impact of failed or delayed product builds isn’t only financial. There are significant non-monetary implications for teams and organizations, including decreased team morale and focus, diminished stakeholder confidence and interest, customer acquisition delays and their resultant poor feedback, and products falling behind the competition.

Enterprises are continuing to embrace tech and digital as core components of their business at an ever-increasing pace. And, as the investments in these areas increase, enterprises need to be ever more cautious to drive the desired results from these investments. In this article, we will share our learnings from helping multiple clients in their journey to build successful digital products and businesses.

Three impediments we have seen over and over again while executing Product Builds:

Being aware of these three impediments will help the owners of digital products at enterprises, from product leaders and product managers to delivery leaders and product owners, to avoid the most common traps which can lead to inefficient digital builds and proactively find ways to mitigate them. Let’s dive in.

1. Unclear requirements: This is one of the most common causes for delays which affect the success of a product launch. Poor clarity on requirements can sprout from multiple sources, including:

• Product vision clarity: If not articulated, socialized, or aligned with stakeholders, this can lead to frequent conflicts and misunderstandings on scope and context. To mitigate this, share and discuss the overall product vision whenever a new team member/stakeholder is onboarded. The product vision can be articulated through various methods, as simple as filling a future press release template or product canvas. A strong sign that the vision is understood comes when team members ask questions such as — ‘How do you think this feature helps us achieve our vision?’
• Functional requirement — quality assurance and management: Poor quality requirements often include unqualified assumptions and incorrect or insufficient details needed to execute a requirement. This leads to a lot of rework and developer frustration. A Definition of Ready (DoR) could help mitigate this. The entire team should align on a DoR that should be met before development begins, including everything from the format of requirement (Gherkin vs. Regular) and design readiness to developer input and stakeholder sign-off. Sometimes, different developers are working on multiple versions of the same requirement. Do you get questions from your development team such as — “But Figma shows this user journey, so that’s how I have implemented it!” To handle this, you should have a single source of truth using a backlog management tool that suits your requirements and set up processes to ensure that changes are curtailed and clearly communicated. These are just some good practices to ensure that the requirements are aligned before the development team starts on that piece of coding and the teams should feel free to find ways that help them stay aligned.
• Technical requirements’ consideration: It is key to consider and manage a product’s technical requirements to avoid any unpleasant surprises. There are two key aspects here. The first is quality, including code quality, security aspects, performance, etc. The likes of coverage and code smells for the quality aspects should be defined and agreed upon in a ‘Definition of done’ with the stakeholders. Any story should be marked as done only if it meets all the quality bars. The second aspect is to ensure a technical runway supporting the development of the functional elements. These tech requirements often involve multiple teams so should be replicated and prioritized across the backlogs of other teams. Time should also be built in to convince and influence these other team members (more on this in later articles).

2. Poor planning: The lack of a plan is the beginning of software development going wrong and is akin to accepting failure. As they say, fail to prepare, prepare to fail. Irrespective of the uncertainties and moving parts in the product development process, planning is essential to ensure that all stakeholders and team members know what they are doing, when they are supposed to do it, and who is responsible for what. Some key aspects to consider while planning includes the following:

• Iterative Planning: When your leadership asks, ‘When can we go live with this?’ the answer should not be, ‘Development will be complete on this date, marketing will then produce a plan on this, and then…’. You should have a clear, up-to-date, comprehensive plan to share with leadership regularly and ad-hoc. A plan should not be limited to development sprints; it should cut through all key workstreams/functions needed to make the product live i.e., customer service setup, marketing communication, design, etc. By comprehensive planning, we do not mean trying to have every detail about the product requirement before starting. In fact, most products inevitably start from unclear requirements, but having a plan and vision is a must-have so that your team is anchored to the tasks and timelines and continues to iterate on them regularly.
• Metric-driven predictability: Product and program managers should be able to identify potential timeline deviations and plan based on team capacity using metrics and methods like burndown charts, velocity, and capacity. Metrics should be available for the leadership and team to view as needed and should be discussed in Sprint review.
• Launch strategy and success metrics: Defining and implementing these success metrics is essential for continuously striving for product-market fit. The development should ensure the plan and requirements are geared towards achieving the defined metrics. This includes determining success in terms of key metrics when the product/feature is launched, in the short-term and the long term e.g., adoption, activation, conversion, bug reports, customer reviews etc. We can imagine some agile advocates jumping off their chairs and suggesting that this level of planning is anti-agile. We believe that planning (if it is done with the correct input from various dependent teams) does not need to delay the execution. The right amount of planning and the right stakeholders can significantly de-risk execution and timelines, allowing to surface gaps in the plan early on.

3. Siloed operating model: While the agile mindset and Scrum or Kanban form the backbone of the processes, some process areas require deliberate effort to keep the product development process smooth and enable the team members to function effectively. It’s vitally important to consider each of the following to ensure the best possible chance of success for your next product development cycle.

• Co-creating with users: Imagine only receiving user feedback at the end of a build cycle as a part of a standalone UAT (user acceptance testing) and hearing ‘This is not what I meant when I said….’ Not only would this be hugely frustrating after a potentially long and drawn-out development cycle, but this can lead to significant delays, as ignoring these inputs can lead to low adoption later. Mechanisms should be set up to obtain and incorporate user inputs regularly, so they are baked into the sprint cadence from sprint 1. This can be achieved by creating a small panel with existing or potential users. This panel can be invited to demos, and immersion sessions each sprint to get the user’s thoughts on a regular basis.
• Cross-functional, goal-focused collaboration: Team mindset becomes critically important in the setup where new teams are formed to deliver on a mission. It is important for the team to establish a common ground concerning individual working styles, roles, responsibilities, learning goals, and aspirations, with the focus being on the mission that they are set together to achieve. A mechanism should be set up to obtain and incorporate inputs from all related teams/departments (marketing/finance/legal) regularly so that their insights are baked into the product right from the start. For example: Inviting members from different departments during the demos to represent their teams and getting their feedback on what is already built and the upcoming planned implementation.
• Continuous improvement: In the rush to deliver, sometimes the teams skip ceremonies like retrospectives, which enable them to pause, reflect and learn from their mistakes and achievements and accordingly adapt the operating model. Actions should be identified out of each retrospective, along with owners for each action, and the action items should be part of a sprint’s plan and objectives. And all this can be done while keeping the retrospectives fun.

While the context of all organizations and digital builds may differ, identifying and mitigating these impediments early will help you ensure that the digital build goes as planned in delivering the desired outcomes. This will help you save not only monetary costs but also the confidence of your team, stakeholders, and customers.

In addition to being aware of these three common impediments and their mitigation strategies, here are a few key skills to have as a Product Manager for successful product builds: What separates top product managers from the rest of the pack?. We look forward to sharing our learnings from our experience in helping companies with their product builds in a series of articles such as below:

1. Holistic value maximization in product builds.

2. Team aspects of a Product build.

3. Web3 architecture layers and choices.

4. Single vs. multi-app strategy — Imperatives and Implications.

5. Gen AI toolkit for Product Owners

Please do share with us your thoughts and experience on how you have mitigated some of the challenges we have outlined above in your product builds.

Reference:

¹https://professionalprograms.mit.edu/blog/design/why-95-of-new-products-miss-the-mark-and-how-yours-can-avoid-the-same-fate/

Three Impediments to Overcome When Executing Digital Product Builds was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Bent Christensen, Principal Cloud Engineer I, Cloud by McKinsey

This article explores how GraphQL can be used to implement a fully event-driven cloud architecture — in the sense that it is comprised of decoupled scalable services in the backend but also extends to the frontend where events can be received in a pub/sub manner. This avoids a clunky and wasteful polling mechanism or, worse, the need for constant refreshes by the end user to get the latest updates.

Both event-driven architectures and GraphQL are trending technology approaches in modern application development, emerging from different domains and for different reasons, which makes the combination of the two an interesting experiment and uncharted territory.

• An event-driven architecture (EDA) is a pattern that promotes developing small, decoupled services that publish, consume, or route events. This makes it easier to scale, update, and independently deploy separate components of the system.
• GraphQL is an API query language for frontend applications meant to replace traditional REST APIs. GraphQL brings a type system to APIs, a declarative query language to avoid over- and under-fetching, real-time updates, and a mechanism to consolidate multiple data sources into a single, unified data-access endpoint. This provides a better and more consistent frontend and backend developer experience through a separation of concerns approach.

The reason for exploring the combination of an EDA and GraphQL is to create a reusable recipe or application pattern that combines the benefits of both technologies in a coherent single approach for modern application development.

The infrastructure implementation for this article is available as Terraform using native services in AWS and a React app for the frontend application.

Architecture overview

The proposed infrastructure implementation is shown in Figure 1, where a frontend React app can send queries to a GraphQL endpoint provided by AWS AppSync. When the processing has finished, the React app will receive the results asynchronously and in real time through a GraphQL subscription connection.

Notice how the initial query only returns an acknowledgment for accepting the request, but the result of the request is delivered out of band through a separate mechanism.

Behind the GraphQL endpoint is a basic EDA that uses an AWS SQS to queue incoming requests, and then an AWS Lambda function will take requests off the queue and start processing, before publishing the result on an EventBridge bus. At this point, a rule will be triggered to invoke a mutation on the same AWS AppSync API, which can be subscribed to by the React app, in order to receive the final result in real time.

Figure 1: GraphQL endpoint provided by AppSync in front of a basic event-driven backend.

Walk-through

The walk-through of each of the infrastructure components follows the flow of an event initiated by a React app end user (the left side of Figure 1) all the way through the architecture and back to the React app.

End user React app

One of the most popular frontend libraries is React, and for this particular exploration of GraphQL, the Apollo Client v3 library was chosen. This was due to its ability to integrate seamlessly into the React component structure and because it has a rich set of configurable features, such as authorization and state management.

Besides the core data fetching and data modifying mechanisms, the Apollo Client v3 has an advanced state management functionality that can support an offline-first approach. This contributes to keeping the frontend source code consistent and avoids the need for an additional state management layer or library such as Redux.

Alternatives to Apollo Client v3 include the following:

• aws-amplify/api is part of the AWS Amplify ecosystem and provides basic support for queries, mutations, and subscriptions provided by AppSync. It’s a good choice if you are already using the Amplify libraries in your app.
• URQL is a streamlined minimalistic library that is highly extensible and can be used with a variety of JavaScript libraries — not only React but also plain JavaScript and node.js, for example.
• Relay is built for scaling and optimization but is also highly opinionated and has a steeper learning curve compared to Apollo Client.

The configuration of the Apollo Client library in the React app can be seen in Figure 2. It is a bare-bones configuration with in-memory cache and uses the AppSync API key for authentication. Most real-world applications would use JSON web tokens (JWT) with appropriate claims, such as role and tenant, to authenticate and authorize the current user within the AppSync resolvers. But this simpler configuration is perfect to showcase the event-driven flow in a development environment without too much overhead.

Also notice the imported packages aws-appsync-auth-link and aws-appsync-subscription-link, which are provided by AWS to integrate the Apollo Client with AppSync.

import { ApolloClient, InMemoryCache, ApolloLink, createHttpLink } from '@apollo/client'
import { createAuthLink } from 'aws-appsync-auth-link'
import { createSubscriptionHandshakeLink } from 'aws-appsync-subscription-link'

const url = process.env['REACT_APP_APPSYNC_ENDPOINT'] || ''
const region = process.env['REACT_APP_APPSYNC_REGION'] || ''

const auth: any = {
  type: 'API_KEY',
  apiKey: process.env['REACT_APP_APPSYNC_API_KEY'] || ''
  // jwtToken: async () => token, // Used for Cognito UserPools OR OpenID Connect.
}

const httpLink = createHttpLink({ uri: url })

const link = ApolloLink.from([
  createAuthLink({ url, region, auth }),
  createSubscriptionHandshakeLink({ url, region, auth }, httpLink)
])

export const getClient = () => {
  const client = new ApolloClient({
    link,
    cache: new InMemoryCache()
  })

  return client
}

Figure 2: React app GraphQL configuration.

GraphQL API with AppSync

The AppSync configuration supports two flows, namely the incoming initial event from the end-user React app and then a pub/sub style event dispatcher communicating the result back to the React app through a subscription. Let’s start with the simpler one — the configuration of handling the incoming event.

Handling incoming events with AppSync

In the GraphQL schema included in Figure 3, a simple query called sendRequest is defined to handle the incoming event. This could also be a mutation if the request involves changes to the state of the data structure in the backend. The sendRequest query takes an integer as input through the variable data that is used in the backend calculation and a channel name that is used to publish the result to the correct end users.

The immediate result from making a sendRequest query is a RequestResponse object containing the status, which can be true or false according to whether or not the sendRequest query was received correctly. Following this, a messageId, which is the tracking ID, can be used to match the correct reply event in case multiple sendRequest queries are performed close together. The ResponseObject contains additional fields for debugging purposes, but these are not used here.

type Result {
  data: Int
  channel: String
  messageId: String
}

type Mutation {
  pubMsg(data: Int!, channel: String!, messageId: String!): Result
}

type Query {
  sendRequest(data: Int!, channel: String!): RequestResponse
}

type RequestResponse {
  status: Boolean
  SendMessageResponse: TSendMessageResponse
  messageId: String
}

type TSendMessageResponse {
  xmlns: String
  SendMessageResult: TSendMessageResult
  ResponseMetadata: TResponseMetadata
}

type TSendMessageResult {
  MD5OfMessageBody: String
  MessageId: String
}

type TResponseMetadata {
  RequestId: String
}

type Subscription {
  subMsg(channel: String!): Result @aws_subscribe(mutations: ["pubMsg"])
}

schema {
  query: Query
  mutation: Mutation
  subscription: Subscription
}

Figure 3: GraphQL API schema for the event-driven architecture.

The resolver behind the sendRequest query field is taking the data input and placing it on an SQS queue for further decoupled processing (more on this in the next section). The resolver is shown in Figure 4.

Notice how there is no intermediate glue-code Lambda function, but the JavaScript resolver is interacting directly with the queue through the HTTP API by building a HTTP POST request body from the data and channel input parameters in the request template. And then in the response template, it is relaying the result from adding the message on the queue and reusing the SQS generated messageId as the tracking ID for the event throughout the infrastructure.

import { util } from "@aws-appsync/utils"

export function request(ctx) {
  const { args: { data, channel },
        stash: { sqs_queue_url },
  } = ctx;
  const MessageBody = util.urlEncode(JSON.stringify({ data, channel }));
  return {
    method: "POST",
    resourcePath: "/",
    params: {
      headers: { "Content-Type": "application/x-www-form-urlencoded" },
      body: `Action=SendMessage&Version=2012-11-05&QueueUrl=${sqs_queue_url}&MessageBody=${MessageBody}`,
    },
  };
}

export function response(ctx) {
  const { result: { statusCode, body }} = ctx;

  const resultMap = {};
  if (statusCode === 200) {
    resultMap["status"] = true;
    resultMap["SendMessageResponse"] = util.xml.toMap(body).SendMessageResponse;
    resultMap["messageId"] =
      util.xml.toMap(body).SendMessageResponse.SendMessageResult.MessageId;
  } else {
    resultMap["status"] = false;
  }
  return resultMap;
}

Figure 4: JavaScript Resolver for the sendRequest query field.

The Terraform configuration of the JavaScript resolver from before is displayed in Figure 5, where the file add_to_sqs_resolver.js is listed in Figure 4. All resolvers using the APPSYNC_JS runtime must be of type pipeline resolver instead of just the unit resolver type.

Notice how the URL of the SQS queue is injected into ctx.stash by Terraform, so other resolvers in the same pipeline can fetch it easily as an environment variable.

resource "aws_appsync_resolver" "add_to_sqs_resolver" {
  type   = "Query"
  api_id = aws_appsync_graphql_api.this.id
  field  = "sendRequest"
  kind   = "PIPELINE"
  code   = <<EOF
    export function request(ctx) {
      ctx.stash.sqs_queue_url = '${aws_sqs_queue.this.url}'
      return {}
    }
    export function response(ctx) {
      return ctx.prev.result;
    }
EOF

  runtime {
    name            = "APPSYNC_JS"
    runtime_version = "1.0.0"
  }

  pipeline_config {
    functions = [ aws_appsync_function.add_to_sqs_function.function_id ]
  }
}

resource "aws_appsync_function" "add_to_sqs_function" {
  api_id      = aws_appsync_graphql_api.this.id
  data_source = aws_appsync_datasource.queue.name
  name        = "add_to_sqs_event_queue"
  code        = file("./add_to_sqs_resolver.js")

  runtime {
    name            = "APPSYNC_JS"
    runtime_version = "1.0.0"
  }
}

Figure 5: Terraform configuration of the JavaScript pipeline resolver for the sendRequest field.

Handling outgoing events with AppSync

Once the calculations in the backend have been successfully performed, you might want to output the result through the AppSync API.

A mechanism to accomplish this pattern is to create a pub/sub style API with a mutation as the publisher and a subscription listening to the mutation as the subscriber, as seen in Figure 6. The mutation is only meant to be called by an internal cloud part of the infrastructure, with the result as a parameter, and the subscription is to be used by the React app to receive the result in real time through an established connection. So, in the React app, before sending any queries to the AppSync sendRequest field, a subscription should be created and established to receive any results of the request.

In the GraphQL schema in Figure 3, the corresponding mutation is called pubMsg and the subscription is called subMsg. The pubMsg mutation takes data, channel, and messageId as parameters, where data is the actual result we want from the request, channel is our filter to ensure only relevant end users receive the result, and, finally, the messageId matches the received result with a sendRequest query within the React app. Our internal part of the infrastructure (big reveal — the EventBridge) will provide these parameters to the pubMsg mutation (detailed later in the section “Relay results with EventBridge”), and the same parameters will automatically also be broadcasted over any established subscriptions meeting certain criteria.

Figure 6: Detailed AppSync event flow.

The Terraform configuration as seen in Figure 7 for the pubMsg resolver is straightforward, since it is used only to relay information and not to perform any computation or transformation. And the subscription subMsg field doesn’t even have any Terraform configuration but is using AppSync default resolvers. From the schema in Figure 4, we can see that the subMsg subscription takes one input parameter when the React app creates a new connection, and that is the channel name.

Notice the not null requirement with the String! type on the channel parameter. This is specific to AppSync and means that before sending anything to listening applications, the result is filtered by requiring a matching value in a field with the name channel in the result object to be sent. Without the ! at the end of the String type in the schema, a React application subscribing without providing a channel name would receive all results, including the results meant for other end users. This is not ideal if any of the results are confidential, but it is also just bad practice to over-fetch data that is not needed. So, filtering by a channel or topic before sending the results over the internet is preferred. In cases where sharing of the results is intended, a pre-shared channel name could be used in the example app, so groups of end users would receive the results whenever one of them makes a new sendRequest query.

For the example prototype developed for this article, a pre-shared channel name works fine to showcase the basic functionality of filtering, but for real-world applications, it is best practice to implement filtering for private sessions on unique end user information such as userId, role, or tenantId claims in a JWT.

resource "aws_appsync_resolver" "pubMsg_resolver" {
  type   = "Mutation"
  api_id = aws_appsync_graphql_api.this.id
  field  = "pubMsg"
  kind   = "PIPELINE"
  code   = <<EOF
    export function request(ctx) {
      return {}
    }
    export function response(ctx) {
      return ctx.prev.result;
    }
EOF
  runtime {
    name            = "APPSYNC_JS"
    runtime_version = "1.0.0"
  }

  pipeline_config {
    functions = [ aws_appsync_function.passthrough.function_id ]
  }
}

resource "aws_appsync_function" "passthrough" {
  api_id      = aws_appsync_graphql_api.this.id
  data_source = aws_appsync_datasource.none.name
  name        = "passthrough"
  code        = <<EOF
    export function request(ctx) {
     return { payload: ctx.args }
    }
    export function response(ctx) {
      return ctx.result;
    }  
EOF

  runtime {
    name            = "APPSYNC_JS"
    runtime_version = "1.0.0"
  }
}

Figure 7: Resolver for the pubMsg mutation.

Queuing incoming events with SQS

The queue for incoming events is following a standard EDA pattern as outlined in the AWS-provided overview of EDA.

As seen in Figure 8, the definition of the event queue itself is simple. I have included the triggering mechanism as well for Lambda to pick up one item from the queue when a new message is added. This is convenient for the example prototype, but in a real-world situation, the processing component of an event-driven architecture might be an on-premises legacy system that is capable of processing a message only every ten minutes. If that’s the case, then the queue will serve its main purpose as part of the architecture and act as a buffer, and the mechanism for consuming messages from the queue will be a pull-based instead of a push-based trigger, as shown in Figure 7.

Further best-practice configurations of the event queue in a real-world scenario include the following:

• Add a dead-letter queue for messages that failed to be processed.
• Use multiple queues to prioritize the order of message processing.
• Optimize consumption of messages from the queue by consuming batches instead of just one message at a time if supported by backend processing.

### Sqs ###
resource "aws_sqs_queue" "this" {
  name = "event_queue"
}

### Lambda to be triggered by sqs ###
resource "aws_lambda_event_source_mapping" "this" {
  batch_size       = 1
  event_source_arn = aws_sqs_queue.this.arn
  enabled          = true
  function_name    = aws_lambda_function.processing_lambda.arn
}

Figure 8: Configuring incoming event queue with SQS.

Since this article is focused on the GraphQL frontend, I wanted to share the configuration seen in Figure 9 of the SQS queue as a DataSource in the AppSync API definition, as this is not an obvious configuration. Notice how the DataSource is of the generic HTTP type, since SQS queues are not directly supported as a DataSource type in AppSync, as are other AWS services such as DynamoDB, Lambda, or RDS.

resource "aws_appsync_datasource" "queue" {
  api_id           = aws_appsync_graphql_api.this.id
  name             = "sqs_event_queue"
  service_role_arn = aws_iam_role.appsync_datasource_role.arn
  type             = "HTTP"

  http_config {
    endpoint = "https://sqs.${var.region}.amazonaws.com/"
    authorization_config {
      authorization_type = "AWS_IAM"
      aws_iam_config {
        signing_region       = var.region
        signing_service_name = "sqs"
      }
    }
  }
}

Figure 9: SQS queue as AppSync DataSource.

This also means that the authorization mechanism must be explicitly configured from outside the AWS console user interface, where there are no fields for specifying IAM roles or policies for accessing the HTTP endpoint, as illustrated in Figure 10.

Figure 10: Adding a new HTTP DataSource for AppSync.

So, for the AppSync resolvers to have enough permissions to add new messages to the SQS queue, a new role that can be assumed by the AppSync service must be created and a policy must be attached to this role, allowing the sqs:SendMessage action as shown in Figure 11.

resource "aws_iam_role_policy" "appsync_datasource_role_policy" {
  name = "appsync_datasource_role_policy"
  role = aws_iam_role.appsync_datasource_role.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "sqs:SendMessage"
      ],
      "Effect": "Allow",
      "Resource": "${aws_sqs_queue.this.arn}"
    }
  ]
}
EOF
}

Figure 11: AppSync DataSource policy for adding a message to SQS queue.

Processing with Lambda

The actual processing component of the event-driven architecture used here to explore GraphQL frontend interaction is merely a placeholder for a possibly complex and sophisticated computation or interaction with an on-premises system. In a real-world situation, the processing component ideally should not be constrained in an event-driven architecture to reply within a certain time limit, such as a HTTP timeout of the browser, but instead be decoupled and publish the reply to any interested subscribers.

The processing with Lambda in the example provided here is simply taking an integer as input from the SQS queue when triggered and then calculating the squared value before publishing the result to EventBridge, as seen in Figure 12.

The messageId field generated by the SQS queue when the event was first received from the React app and returned immediately to be used as the tracking ID of the request is now pulled out of the input event coming from SQS to the processing Lambda and is injected back into the Detail field of the message before being published on EventBridge. This allows the React app to track and match the messageId value received in the final result message with the initial sendRequest tracking ID received earlier.

const AWSXRay = require('aws-xray-sdk-core')
const AWS = AWSXRay.captureAWS(require('aws-sdk'))
AWS.config.update({ region: process.env.APPSYNC_REGION })

const eb = new AWS.EventBridge()

const legacy_computation = (input) => {
  // return the value squared...
  return input * input  
}

exports.handler = async (event) => {
    // Response object
    const res = { data: 0, channel: '', messageId: '' }

    // Take the next item on the event_queue
    if (event.Records && event.Records[0].body && event.Records[0].body.length > 0) {
        const req_args = JSON.parse(event.Records[0].body)  

        // Perform some time-consuming computation or interaction with other systems.
        res.data = legacy_computation(req_args.data)
        res.channel = req_args.channel
        res.messageId = event.Records[0].messageId
    }

    // Publish the result on eventBridge
    const params = {
        Entries: [{
            Detail: JSON.stringify(res),
            DetailType: 'gql-eda-poc',
            Source: 'com.ticketdispenser.gql-eda-poc',
            EventBusName: process.env.EVENTBUS_NAME
        }]
    };

    const log_result = await eb.putEvents(params).promise()
}

Figure 12: Processing Lambda publishing reply on EventBridge.

Relaying results with EventBridge

The heart of any event-driven architecture usually contains an event router or a pub/sub broker where events can be relayed between the decoupled architecture components. For this prototype, EventBridge is used as the event router for the result of the processing Lambda described in the previous section.

Creating the EventBridge resource is the easy part, as seen in Figure 13, where it consists of three lines of configuration that provide a name for the EventBridge resource as the only parameter. And then an EventBridge rule is added to match messages sent from the processing Lambda by the content of the DetailType field of the message (gql-eda-poc).

### EventBridge ###
resource "aws_cloudwatch_event_bus" "this" {
  name = "gql-eda-poc-messages"
}

### EventBridge rule ###
resource "aws_cloudwatch_event_rule" "this" {
  name           = "send-to-gql"
  description    = "Send event to appSync"
  event_bus_name = aws_cloudwatch_event_bus.this.name

  event_pattern = <<EOF
{
  "detail-type": ["gql-eda-poc"]
}
EOF
}

Figure 13: EventBridge and API destination configuration.

The next step in the event flow is to return the message back to the pubMsg mutation in the AppSync API, so the subscription subMsg will send out a message to the React app and the flow is complete. EventBridge does not yet support AppSync as a native AWS service target for EventBridge rules, so, instead, the generic HTTP API destination target type can be used, as shown in Figure 14. The authentication for AppSync also needs to be manually configured by adding the AppSync API KEY to the header of the HTTP POST request made from the EventBridge service to the AppSync service.

Alternatively, a Lambda function could be used as an intermediary step, since Lambda is supported natively as a EventBridge target, and then the Lambda function could turn around and call the AppSync pubMsg using the standard AWS SDK library. This would work, and in some cases might be necessary, such as when an AppSync API KEY is not available or more fine-grained IAM-based access control is needed. Though, unless a flow step is essential, best practice is to keep the happy path as lean as possible and cut out any unnecessary steps.

### GQL API destination ###
resource "aws_cloudwatch_event_api_destination" "this" {
  name                = "gql-pub-destination"
  description         = "Send event to appSync"
  invocation_endpoint = aws_appsync_graphql_api.this.uris["GRAPHQL"]
  http_method         = "POST"
  connection_arn      = aws_cloudwatch_event_connection.this.arn
}

resource "aws_cloudwatch_event_connection" "this" {
  name               = "gql-connection"
  description        = "A connection to appSync"
  authorization_type = "API_KEY"

  auth_parameters {
    api_key {
      key   = "x-api-key"
      value = aws_appsync_api_key.this.key
    }
  }
}

Figure 14: Adding the AppSync mutation as an API destination.

The final resource needed for EventBridge to directly interact with the subMsg mutation on the AppSync API is the EventBridge target that connects the event rule with the API destination and then constructs a HTTP POST mutation request from the information in the EventBridge event. This is not a trivial configuration and can require some tinkering to get just right.

For the example prototype, the configuration can be seen in Figure 15, where data, channel, and messageId are first extracted from the EventBridge event as received from the processing Lambda and then plugged back into the template of a GraphQL pubMsg mutation request as it looks on the HTTP layer and finally it will be invoked on the AppSync API.

resource "aws_cloudwatch_event_target" "this" {
  rule           = aws_cloudwatch_event_rule.this.name
  target_id      = "SendToAppSync"
  event_bus_name = aws_cloudwatch_event_bus.this.name
  arn            = aws_cloudwatch_event_api_destination.this.arn
  role_arn       = aws_iam_role.eb_role.arn

  input_transformer {
    input_paths = {
      "data" : "$.detail.data",
      "channel" : "$.detail.channel",
      "messageId" : "$.detail.messageId"
    }

    # Needs to be a one-liner...
    input_template = <<EOF
{ "query": "mutation PubMsg($data:Int!,$channel:String!,$messageId:String!){ pubMsg(data:$data,channel:$channel,messageId:$messageId){ data, channel, messageId }}", "operationName": "PubMsg", "variables": {"data": <data>, "channel":"<channel>", "messageId":"<messageId>" }}
EOF
  }
}

Figure 15: Transforming an EventBridge event into an AppSync mutation HTTP request.

Coming full circle

This concludes the walk-through of the event flow through the architecture components with an emphasis on the end-to-end real-time event flow including the frontend by using GraphQL to support backend requests and frontend notifications asynchronously.

The Terraform configuration of certain resources was also highlighted and discussed during the walk-through, especially in situations where the configuration wasn’t obvious or non-trivial, such as configuring an SQS queue as a generic HTTP AppSync DataSource with corresponding roles and policies and configuring an AppSync mutation as a generic HTTP API destination target with authentication HTTP headers in EventBridge. Hopefully, some of these approaches and configurations can be reused or refined further by others implementing an EDA with a GraphQL frontend.

Wrap-up

Event-driven architectures and GraphQL APIs are both independently exciting emerging approaches for the development of modern, flexible, and scalable cloud applications and APIs. This article has combined the two and explored the hypothesis that we can achieve the benefits of both in a single approach.

And, indeed, the combination looks to be a promising application pattern or reusable recipe to keep handy in the cloud developer’s toolbox, despite not being fully natively supported by all the AWS services — workarounds using generic HTTP destinations have been outlined to bridge the gaps.

Toward end-to-end real-time event-driven architectures using GraphQL with AWS AppSync was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Ruth Tupe, Expert Associate Partner, Build by McKinsey

In March, we were lucky enough to not only attend the IxDA conference but host a workshop on inclusive design, where we spoke about an opportunity we had to drive holistic impact in design in the context of government services. This workshop explored a new concept of how inclusivity is not just a social imperative, but also a strategic way to build more resiliency into products and services.

Over the past few years, the accelerated pace of technological change combined with external forces to present a set of new challenges that force us to re-evaluate our systems and in turn, how to design them. From the pandemic to population growth to the rise of AI — the world as we know it is dramatically changing. These changes offer us up the opportunity to redesign out systems with inclusivity built in; it’s our job as designers to work with these changes when we’re designing, rather than against them:

• An aging population: As of 2020, almost 50% of people are living to be 65-74 years old in the US. With life expectancy higher than ever, this population offers up a massive potential workforce with a high level of professional knowledge and expertise. However, an aging population puts a strain on healthcare and the social welfare system.
• Growing socioeconomic divide: The COVID-19 pandemic highlighted economic inequality on a global scale. The pandemic disproportionately affected those living in poverty and while many countries were plunged into economic turmoil, the richest got richer. In fact, research carried out by Oxfam shows that while 99% of humanity is financially worse off since the pandemic, the wealth of the world’s 10 richest men has doubled, widening the economic divide.
• Technological progression: 35% of all businesses now report using AI as part of their day-to-day operations. Whilst some fear certain roles may get replaced by automation, it could drastically reduce the time employees spend on menial tasks, meaning businesses will have more capacity to do more complex work.
• Climate change: By now, most of us are aware of the effects of climate change but few know about the financial implications. Data shows that in the last five years, 89 climate disaster events cost $595.5B in the US alone. Now we are faced with the challenge of designing infrastructure that is resistant and able to withstand more extreme weather.
• Hybrid working: The COVID-19 pandemic created a new normal — ‘hybrid working’. A recent study showed that 38% of employees prefer this way of working and it’s not hard to see why with less commute time and better work-life balance. But it’s not without its issues; less face time with colleagues leads to a lack of connectivity in the workplace.

As these shifts occur, they can often disrupt our systems and widen the inequality gap. This leads to economic, social and political polarization, with the most disadvantaged populations bearing the brunt. For example, research shows that since the pandemic, the average income of those in the bottom 20% of earners has fallen by 6.7%.

Resiliency is a new mindset that we need to adopt. These unpredictable, novel disruptions are affecting every part of the way we live, work, and interact with the world. To deal with this we need to build resilience into every product and system we design so they can withstand shocks or surprises.

How do we become resilient to things we can’t even predict? The key is to consider inclusivity in everything we do. Traditionally, inclusivity is addressed by companies through a DEI initiative, meeting basic accessibility web requirements, or seen as a social imperative. But another way to think about inclusivity is that it puts resiliency into the fabric of solutions. When we create designs that serve a multitude of needs, for a wide array of users, that creates a more resilient design by nature. By considering a wide range of factors, use cases, and devices, you can integrate resilience into your designs from the off, rather than waiting to be reactive to a trigger.

So, how do we practice inclusive design?

First of all, it’s important to define what we actually mean. Inclusive design aims to build equitable experiences that serve the range and diversity of human backgrounds such as ability, age, culture, identity, and language persistently over time. We want to emphasize the longevity of a solution which is often overlooked. Inclusive experiences can only be possible by creating a system and tools to maintain that quality. When a new device, channel, or user group is introduced to your product or experience, are they still able to access it with the same experience parity?

This is not only important from an ethical standpoint, but from a business one too. There are a huge number of underserved communities in existence that represent a great deal of purchasing power; some experts put the figure as high as $12 trillion or more.

Over the past two years we’ve developed a set of principles that enable resilient, equitable and inclusive design:

• Broad: When designing, you must consider a wide range of users and be mindful of intersectionality.
• Deep: Do the work to really understand the experiences and unmet needs of potential users, including underlying systems of bias in play.
• Participatory: Invite users to the design process, designing “with” rather than designing “for”.
• Proactive: Plan for change. Anticipate unintended consequences by thinking through challenging “what-if” scenarios and prioritise building in the flexibility to adapt to changing circumstances.
• Measurable: Detect change. Listen for the earliest signals of harm or other unintended consequences. Build in the ability to measure and quantify from the beginning.
• Iterative: Enable solutions to evolve in response to changing user needs and circumstances by building a set of levers to adjust your solutions

Inclusive design in government services

One area where inclusivity and resiliency matter most is in government services. As a sector it makes a great case study because it is constantly under pressure to remain operational, regardless of what else is going on in the world — and the penalty for mistakes is high. For this reason, building resilience into government systems is integral and applying inclusive design principles is the best way to achieve this.

One great example is the design of Governor’s Island in New York City. The government knew that its coasts were vulnerable to storms and hurricanes so it decided to commission landscape architect Adriaan Geuze to design a solution. Understanding the impact that climate change is having on rising water levels, Adriaan knew that it was vital to raise the shore front of the island. In 2007 he proposed building 15 ft high hills along the southernmost part of the island to protect it from storm surges and flooding. In 2012 his design was put to the test when Hurricane Sandy hit New York, causing 200,000 homes to be destroyed and more than $50 billion worth of damage. But despite the extensive damage to Manhattan, Governor’s Island was left unscathed due to Adriaan’s well-designed flood barriers.

https://www.youtube.com/watch?v=ikve-HCo9hw

This project was a main source of inspiration for us, so we applied similar inclusive design principles to an unemployment services journey to see how we can build a more resilient experience for citizens. We went through a 3-step process to analyze the problem with an inclusive lens:

1. Understanding symptoms of bias — Beyond personas, we believe greater empathy lies in taking a moment to reflect on potential prejudices and biases people may encounter. To trigger these discussions, we supplemented personas with ‘Ability’ and ‘Identity’ cards to understand intersectionality.
2. Generate inclusive interventions with clear ROI — We encouraged participants to not just ideate but to consider how their solution would bring business value to the organization.
3. Test the resiliency of your solutions — After participants created solutions, we did future scenario planning by getting participants to draw trend cards to see if their solutions would still survive (e.g., if a hurricane happened, would you solution still work)

Overall, participants came up with innovative solutions to re-imagine government services that they never would have thought of before. One of the most innovative ideas that came out of the workshop was a “dating” app for pairing employees looking for work with employers. The app would allow employees to be matched quickly and efficiently with jobs that are suitable for them. The app would very much be human-centric, allowing employees and employers to ‘date’ one another on a trial basis to make sure they’re a good fit. Another idea that stuck out to us was using an AI coach to guide candidates through the job interview process, allowing them to do practice runs to build up their confidence. The AI would also have the capacity to break down language barriers throughout the job application process by coaching the applicant in their native language. The concepts and ideas generated during this workshop are a great example of how our approach to inclusive design can open the door for creativity and innovation.

Through this structured approach, we reframed inclusivity not as feel-good initiative but a strategic play with business case rationale. Too often, inclusivity initiatives are not taken seriously and deprioritized because it’s seen as a compliance activity rather than a long-term investment. It’s time to shift the conversation on what inclusivity means and utilize it properly to maximize its impact both socially and in business.

—

Additional reading:

https://www.ibm.com/watson/resources/ai-adoption

https://www.brookings.edu/research/what-the-2020-census-will-reveal-about-america stagnating-growth-an-aging-population-and-youthful-diversity/

https://climate.nasa.gov/evidence/#otp_evidence

https://www.gallup.com/workplace/390632/future-hybrid-work-key-questions-answered-data.aspx

Inclusive Design as a path to resiliency and growth was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Mohamad Assaf — Senior Analyst, Build by McKinsey; Brian Leke — Senior Expert and Associate Partner, Build by McKinsey; and Sallah Kokaina — Senior Expert and Associate Partner, Build by McKinsey

Offering fast and high-availability solutions is becoming the new norm in the field of software engineering. With the growing demands on web and mobile channels, developers can leverage open-source solutions to shape production-ready setups.

During our own practice, we have found it valuable to adopt a ‘shift-left’ approach that puts performance concerns at the core of development activities. This is especially important when it can affect end-user experience and minimize the impact of growing digital initiatives.

In this article, we will explain how implementation can be achieved on a high-availability container cluster using OpenShift to scale a Redis cache consumed by a sample microservice written with Spring Boot. Above all, we will help you avoid the pitfalls of data inconsistencies that can occur when using caching.

Using caching can stop system users from being affected by a slow response of downstream components for data that is static, but it often presents the following challenges:

• Time-to-live — An object is cached for a specific period, usually based on the frequency of refresh or change. This requires correct configuration to ensure that data is available for the appropriate period of time before refreshing.
• Performance — Caching is used to speed up response times. Using the right caching practices is key to achieving the best performance.
• Availability — The cache may be used for operational tasks that need a quick response. Ensuring the availability of our cache will prevent any disturbance to our application.

It’s important to bear these issues in mind when integrating a Redis cache with a Spring Boot application.

To kick-start a Spring Boot application, we can start by using spring initializr: https://start.spring.io/

Integration with Redis

We recommend using a Spring Boot starter to integrate with Redis, by adding the spring-boot-starter-data-redis dependency to your project’s pom.xml file. You can also add the dependency if you are using Gradle as a build tool.

Maven

<dependency> 
    <groupId>org.springframework.boot</groupId> 
    <artifactId>spring-boot-starter-data-redis</artifactId> 
</dependency> 

Gradle

implementation 'org.springframework.boot:spring-boot-starter-data-redis:2.2.0.RELEASE' 

Specify the connection details in the application.properties file or the application.yml file using the following properties:

spring.redis.host = <host_url_for_redis>

spring.redis.port = <port_for_redis_instance>

spring.redis.password = <password_for_default_redis_account>

Spring will then leverage boilerplate code to establish the connection. To test the above configuration, you can use the official docker image for redis:

(docker run -p 6379:6379 -d redis:6.0 redis-server — requirepass “testPassword”)

Then add the following configuration in the properties file:

spring.redis.host = localhost

spring.redis.port = 6379

spring.redis.password = testPassword

Most articles for configuring Redis with Spring Boot suggest using the @Cacheable annotation on the main class. While this does automatically configure caching in the application, we recommend implementing caching manually, by using a caching service. This will give us the most control over all the details of our cache, such as control TTL and the structure of the Redis cache we use (Redis Strings, Redis Hashes, etc.) Below is an example of our service:

@Component 
@RequiredArgsConstructor 
public class RedisService { 
    private final RedisTemplate<Object, Object> redisTemplate; 
 
    public <V> void set(String cacheName, String key, V value, long timeout, TimeUnit timeUnit) { 
        redisTemplate.opsForValue().set(cacheName+key, value); 
        redisTemplate.expire(cacheName+key, timeout, timeUnit); 
    } 

 

    public <V> V get(String cacheName) { 
        return (V) redisTemplate.opsForValue().get(cacheName); 
    } 
 
    public Boolean hasKey(String cacheName, String key) { 
        return redisTemplate.hasKey(cacheName+key); 
    } 
} 

For performance gains over the default Java serializer, and to make it easy to inspect and debug the cache in Redis, we configure a custom serializer/deserializer. We will use a default string Serializer/Deserializer for the key, and Jackson to serialize objects. We can then configure our RedisTemplate bean using the below:

@Slf4j 
@Configuration 
@Profile("!test") 
public class RedisConfig { 
    @Bean 
    public RedisTemplate<Object, Object> redisTemplate(RedisConnectionFactory connectionFactory) { 
        RedisTemplate<Object, Object> redisTemplate = new RedisTemplate<>(); 
        try { 
            redisTemplate.setKeySerializer(new StringRedisSerializer()); 
            redisTemplate.setDefaultSerializer(new GenericJackson2JsonRedisSerializer()); 
            redisTemplate.setConnectionFactory(connectionFactory); 
        } catch (Exception e) { 
            log.error("Error getting Redis Template connection ", e); 
        } 
        return redisTemplate; 
    } 

} 

Time-to-live

Redis allows TTL configuration on the cache. If we wish to configure TTL on the entry level, in order to expire a specific object based on a refresh time, then we can use Redis strings, which are simple key-value pairs. On the other hand, if we used Redis hashes, then all fields inside the hash will be deleted when TTL expires.

Take a look at the function set in our RedisService:

  public <V> void set(String cacheName, String key, V value, long timeout, TimeUnit timeUnit) { 
        redisTemplate.opsForValue().set(cacheName+key, value); 
        redisTemplate.expire(cacheName+key, timeout, timeUnit); 
    }

It takes timeout (an integer) and timeUnit (TimeUnit.Hours, or Seconds, as a unit to the timeout) as parameters. When we insert a new entry, RedisTemplate will use the cache as a key-value pair, where the key is the cache name concatenated with the key, and the value will be the serialized class using Jackson. Then, we will set the expiry duration for that cache.

Performance

Referring to https://redis.io/docs/management/optimization/memory-optimization/, we can conclude from the above that a Redis hash map with few fields will behave faster than if it had a few Redis strings. We can demonstrate this with the practical example below.

Consider case 1, a Redis hash called Object:

Object: {

1: test1,

2: test2,

3: test3

}

And case 2, where we have 3 Redis strings with keys Object1, Object2, Object3:

Object1: test1

Object2: test2

Object3: test3

Using case 1 is faster than case 2.

The reason is that Redis transforms the hash map into an array, which leverages CPU cache memory more effectively. So categorizing our entries into smaller hashes will leverage the Redis capabilities and speed up our response time. Here’s an example of how we’ve implemented this technique.

Context:

We had 30,000 customers whose information was cached in Redis. The TTL was 18 hours.

Straightforward caching strategy:

For each client, we had to insert a Redis string (key-value pairs) and cache it, leaving us with 30,000 Redis strings.

Example for a customer id = 100:

customerDetails100: {CustomerDetailsObject}

Leveraging the capabilities of Redis:
In this example a set of customers was mapped into a specific application user. On average, every user had 30 customers. Every time the user logged in, all customer details were fetched (from the cache). We had to structure our cache better to improve the overall performance. We achieved this using Redis hashes:

For a user id = A01, for client IDs 100 to 129

customerDetailsA01: {

100: {CustomerDetailsObject}

101: {CustomerDetailsObject}

102: {CustomerDetailsObject}

…

}

Overall, the set of 30,000 clients was divided into 1,000 sets of 30 clients each, and since all the set is filled automatically when the user logs in, we could configure TTL on the level of the Redis hash.

Availability

Redis may become an important system in our application for speedup and perhaps for critical operations like invalidating JWT tokens or responding to brute-force login.

In this case, when Redis goes down, it would make our application unavailable, so we have to ensure the availability of Redis. Below is a brief explanation on how we can avoid this in the two links below:

Redis provides a setup for high availability, using the Redis Sentinel https://redis.io/docs/management/sentinel/

For a way to deploy Redis to Kubernetes

https://www.containiq.com/post/deploy-redis-cluster-on-kubernetes

We used OpenShift to host our application. In this context, we can deploy Redis as a StatefulSet with two replicas. On startup, we can configure one of the instances to be a replica of the other. We can deploy Sentinel using the following:

apiVersion: apps/v1 
kind: StatefulSet 
metadata: 
  name: sentinel 
spec: 
  serviceName: sentinel 
  replicas: 1 
  selector: 
    matchLabels: 
      app: sentinel 
  template: 
    metadata: 
      labels: 
        app: sentinel 
    spec: 
      initContainers: 
      - name: config 
        image: 10.34.174.71:8082/redis:7.0.5-alpine 
        resources:  
          limits: 
              cpu: "200m" 
              memory: "256Mi" 
          requests: 
              cpu: "100m" 
              memory: "128Mi" 
        command: [ "sh", "-c" ] 
        args: 
          - | 
            REDIS_PASSWORD=<password> 
            nodes=redis-0.redis,redis-1.redis,redis-2.redis  
            for i in ${nodes//,/ } 
            do 
                echo "finding master at $i" 
                ROLE=$(redis-cli --no-auth-warning --raw -h $i -a $REDIS_PASSWORD info replication | awk '{print $1}' | grep role: | cut -d ":" -f2) 
                if [ "$ROLE" == "master" ]; then 
                    echo "found master at $i" 
                    MASTER=$i 
                    break 
                else 
                    echo "no master found" 
                    MASTER= 
                    break 
                fi 
            done 
            echo "sentinel monitor mymaster $MASTER 6379 1" >> /tmp/master 
            echo "port 5000 
            sentinel resolve-hostnames yes 
            sentinel announce-hostnames yes 
            $(cat /tmp/master) 
            sentinel down-after-milliseconds mymaster 5000 
            sentinel failover-timeout mymaster 60000 
            sentinel parallel-syncs mymaster 1 
            sentinel auth-pass mymaster $REDIS_PASSWORD 
            " > /etc/redis/sentinel.conf 
            cat /etc/redis/sentinel.conf 
        volumeMounts: 
        - name: redis-config 
          mountPath: /etc/redis/ 
      containers: 
      - name: sentinel 
        image: 10.34.174.71:8082/redis:7.0.5-alpine 
        resources: 
          limits: 
            cpu: "200m" 
            memory: "256Mi" 
          requests: 
            cpu: "100m" 
            memory: "128Mi" 
        command: ["redis-sentinel"] 
        args: ["/etc/redis/sentinel.conf"] 
        ports: 
        - containerPort: 5000 
          name: sentinel 
        volumeMounts: 
        - name: redis-config 
          mountPath: /etc/redis/ 
        - name: rdata 
          mountPath: /data 
      volumes: 
      - name: redis-config 
        emptyDir: {} 
      imagePullSecrets: 
        - name: regsecret           
  volumeClaimTemplates: 
  - metadata: 
      name: rdata 
    spec: 
      accessModes: [ "ReadWriteOnce" ] 
      storageClassName: "standard" 
      resources: 
        requests: 
          storage: 1Gi 

In the Init container, Sentinel will scan the Redis instances, find the master and monitor it. It will automatically discover the replica and will promote it to master anytime the master is unavailable.

Conclusion

Configuring effective caching can be a challenging task and when not done properly can lead to bugs, such as outdated data and reduced performance. We have provided a way to approach caching that satisfies business requirements regarding frequency of refresh, and availability to prevent system outages. We have also demonstrated how to use the intrinsic capabilities of Redis to improve performance through the structure of cached data. Follow the advice laid out in this article to speed up your implementation and application responsiveness. Please refer to the full source code and Readme on: https://github.com/mohamadassaf96/spring-redis-poc

Effective and highly available cache: Integrating Spring Boot with Redis was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Chris Do — Senior Cloud Architect, and Adrian Cortio — Senior DevOps Engineer II

As you have no doubt seen, companies have already started to run critical workloads with sensitive information on public cloud platforms. Doing this requires building new types of security capabilities — ones that automate the interrogation of cloud systems against security policies. This article, the fourth in a series on deploying Golden Images for GCP, dives into how to use vulnerability scanning tools to ensure systems running in GCP adhere to your company’s security policies. We will dive into how to integrate a vulnerability scanning tool that can be consumed by your Golden Image Pipeline to ensure that the images being created meet a certain security threshold. This article is not a guide or discussion on Software Composite Analysis (SCA), Static Application Security Testing (SAST), or Dynamic Application Security Testing (DAST).

Below are the other articles in this series:

• Part 1: Deploy Golden Images to GCP using Packer and Cloud Build
• Part 2: Update GCP Managed Instance Groups using Cloud Asset Feeds
• Part 3: Use GCP Security Command Center’s Security Findings to track GCE disks that are using old Golden Images
• Part 4 (this article): Integrating vulnerability scanning into your Golden Image Pipeline in GCP

What is a vulnerability scanning tool?

A vulnerability scanning tool is used to investigate a target to look for known vulnerabilities, which are typically published as CVEs (Common Vulnerabilities and Exposures). These vulnerabilities are tied to a score that indicates the potential risk if the vulnerability were to be exploited. There are many different types of vulnerability scanning tools, but this article focuses specifically on tools that look at vulnerabilities in virtual machine images. These types of host-based scanners can look for operating system misconfigurations, outdated software patches, and sometimes network-related vulnerabilities. The vulnerability scanning tool will typically produce a report that outlines all vulnerabilities on the system and information about the vulnerability, such as the severity, date discovered, description, and remediation recommendations.

Why should I use a vulnerability scanning tool in my Golden Image Pipeline?

Using Packer allows us to embed some controls into the image creation process, as this is captured in code similar to infrastructure as code (IaC). However, this approach lacks the ability to inspect for vulnerabilities. Our solutions complement the use of Packer to ensure that all images generated are free of any critical or high-level vulnerabilities and consequently automatically fail image builds that don’t meet organizational security standards. By automating the approval of the image build process, we reduce the cognitive load and management overhead for security and operations teams. The vulnerability scanning tools we chose to use are Trivy (open source) and Tenable.io (commercial-off-the-shelf SaaS), but there are numerous vulnerability scanning tools that perform similarly that can be used with our architecture.

How does the scan interact with images?

There are various scanning methods, depending on the tool, networking, cloud provider, operating system, and access restrictions. These methods can be broken into two major categories: “agent” and “agentless.” Agent scans require a small piece of software to be installed or run on the target system. The agent is able to communicate with the management server and receive updates and scan operations. Agent scans don’t require any sort of credentials to perform privileged access scans. Agentless scans have several subcategories. We will be using a credentialed scan, and there is no software installed on the target system, making it a more lightweight solution. This means a scanner (container or VM) uses credentials to log in remotely to the target and run a set of checks. Credentialed scans require the target to have the proper ports open and our credentials to be configured as administrators so we can run elevated checks. Both approaches have their own advantages and disadvantages. We will be using both of these methods.

How to integrate the solution into your pipeline

There are many CI/CD tools out there that can be used to create your Golden Image Pipeline, such as Jenkins, GitLab, GitHub Actions, GCP Cloud Build, AWS CodeStar, etc., but this article will abstract the pipeline tooling and focus on the vulnerability scanning solution in GCP that will be consumed by the pipeline. There are multiple architecture solutions that can be designed to integrate a vulnerability scanning tool into a pipeline, but for our use case, we opted to use Google Cloud Functions tied together with Google Cloud Workflows. Workflows allows you to create an event-driven architecture that connects various services together. Workflows provides us the ability to initiate a Workflows execution from a gcloud CLI command and provide a JSON payload that includes parameters to configure and initiate the vulnerability scan.

There are multiple vulnerability scanning tools available, and for this article, we opted to use a commercial-off-the-shelf (COTS) product (Tenable.io) and a free open-source product (Aquasec Trivy) to showcase two different options for scanning images.

Solution architecture — Tenable

The diagram below depicts our Tenable vulnerability scanning solution from the invocation of the Google Workflows from the Golden Image Pipeline all the way to the completion of the Workflows, which returns a response back to the Golden Image Pipeline. The entire process is outlined below. Tenable.io is a COTS SaaS product that is used by many companies for their vulnerability scanning, so we felt it was a good product to demonstrate. The typical Tenable/Nessus scanning configuration involves a scanner instance that is managed by the Tenable.io console, it and will perform scans on target instances based on a scan configuration.

1. Golden Image Pipeline

The Golden Image Pipeline will create the image, as documented in the previous articles in this series. After the image is created, a new step in the pipeline includes invoking the Google Workflows to initiate the scanning process.

2. Request payload

In order to invoke the Google Workflows, we use a simple gcloud CLI command, which is shown below. This command is executed from the Golden Image Pipeline runner. It includes parameters that are required for the Google Workflows to execute and configure the scan, such as the recipient of the scan results, the image that needs to be scanned, the OS type, and the instance size.

3. Google Workflows

The Google Workflows execution begins and uses the data parameters provided to kick off a series of Cloud Functions. These Cloud Functions are serverless compute resources that run snippets of code, with each one taking input request data and returning response data. The outputs of one Cloud Function are the inputs for the next Cloud Function in the Workflows execution. The Workflows output is configured in a YAML document — an example is shown below. One of the steps includes initiating a sub-Workflow called poll_workflow, which acts as a polling mechanism to check on the status of the vulnerability scan, since it can take around an hour. The poll_workflow continues looping until the scan is complete.

4. Google Cloud Functions

There are many Cloud Functions that are used to complete the Workflows, which are detailed below:

• create_target_vm: This function receives the Workflows execution payload, including the image, type, operating system, and contact email. It then creates a target VM using the input parameters that will be scanned by the Nessus Scanner.
• initiate_scan: This function receives the response from the create_target_vm function, including the instance name, instance IP address, operating system, and contact email. These values are then used to initiate a connection to the Tenable.io API, which then starts the scan using our Nessus Scanner VM that is located in the same subnet as the newly created target VM. This generates a scan_id, which is used to check the status of the scan and retrieve the report when it’s completed.
• poll_function: This function receives the response of the initiate_scan function, which includes the scan_id. This function then initiates a Tenable.io API call to check the status on that scan_id. The function then returns the status of “complete” or “running” back to the Google Workflows, which will loop back into this function after a short wait period if the status is “running.”
• handle_results: This function receives the response of the poll_workflow sub-Workflow, which includes the scan_id. This function then initiates a Tenable.io API call to retrieve the report on that scan_id. The report is stored in a Google Cloud Storage Bucket in CSV format. This function will also delete the target VM and remove the asset from Tenable.io for licensing purposes. Finally, this function returns the number of vulnerabilities for each severity, as well as the URL to download the CSV from the Storage Bucket.

5. Other resources

There are some supporting resources used in this solution, which are described below:

• Nessus Scanner VM: This is a persistent compute engine instance that is used to scan all of the target instances that contain the image that needs to be scanned. The Nessus Scanner VM communicates with Tenable.io to initiate scans and send results.
• Google Cloud Storage Bucket: This bucket is used to store artifacts and scan results. The artifacts are things such as the Cloud Functions build files, startup scripts, scan results, etc.
• Google Cloud Networking: These are the networking resources required to have the solution running on GCP, such as a virtual network, subnet, firewall rules, Cloud NAT, etc.
• Tenable.io: This is the vendor’s console/management interface that controls the Nessus Scanner VM and also contains all of the data and metrics related to the scans.

6. Response payload

The response payload is returned by the Google Workflows to the caller, which in this case is the Golden Image Pipeline. The response contains the number of vulnerabilities by severity as well as the URL to download the scan report from Cloud Storage. An example response payload is shown below:

7. Sample Tenable.io report

Below is an example scan report of the image we used from the Tenable.io web interface:

Solution architecture — Trivy

The diagram below depicts the architecture for the Trivy vulnerability scanning solution from the invocation of the Google Workflows from the Golden Image Pipeline all the way to the completion of the Workflows, which returns a response back to the Golden Image Pipeline. Trivy is an open-source scanning solution that is quick and easy to get started. One limitation of Trivy is that it only scans Linux instances natively. As you can see, this solution uses fewer Cloud Functions. This is due to the nature of how Trivy works — it’s a simpler architecture. Trivy is downloaded and installed on the target instances, and a simple CLI command is executed on the target instance to execute the scan. The scan output is then saved to Cloud Storage. The entire process is outlined below:

1. Golden Image Pipeline

The Golden Image Pipeline will create the image, as documented in the previous articles in this series. After the image is created, a new step in the pipeline includes invoking the Google Workflows to initiate the scanning process.

2. Request payload

In order to invoke the Google Workflows, we use a simple gcloud CLI command, which is shown below. This command is executed from the Golden Image Pipeline runner. It includes parameters that are required for the Google Workflows to execute and configure the scan, such as the recipient of the scan results, the image that needs to be scanned, the OS type, and the instance size.

3. Google Workflows

The Google Workflows execution begins and uses the data parameters provided to kick off a series of Cloud Functions. These Cloud Functions are serverless compute resources that run snippets of code, with each one taking input request data and returning response data. The outputs of one Cloud Function are the inputs for the next Cloud Function in the Workflows execution. The Workflows output is configured in a YAML document – an example is shown below. One of the steps includes initiating a sub-Workflow called poll_workflow, which acts as a polling mechanism to check on the status of the vulnerability scan, since it can take around an hour. The poll_workflow continues looping until the scan is complete.

4. Google Cloud Functions

There are many Cloud Functions that are used to complete the Workflows, which are detailed below:

• create_target_vm: This function receives the Workflows execution payload, including the image, type, and operating system. It then creates a target VM using the input parameters that will be scanned. This instance is created using a startup script, which is detailed below:

startup.sh

1. Install jq and Trivy: These are packages required for parsing and running the scan.
2. Run Trivy scan: The Trivy package is initiated with a simple CLI command:

3. Retrieve metadata: Metadata is retrieved from the instance to access details about where to upload scans.

4. Parse and upload: The scan results are parsed using jq to find the count of each severity, and then the JSON file is uploaded to Cloud Storage.

5. Delete VM: Finally, the VM deletes itself using the Google Cloud REST APIs for Compute Engine.

• json2csv: This is a simple function that triggers on Cloud Storage Bucket events in the specific Cloud Storage Bucket/Folder and will convert the JSON to CSV. This was added because the jq to CSV feature wasn’t working properly with the dataset provided by Trivy, so we created a Cloud Function to handle this more elegantly. This function also adds unique metadata to the CSV object so that the solution can check for new CSV files that match the Google Workflows execution.
• check_scan_status: This function checks the CSV folder in the Cloud Storage Bucket for new CSV files that have metadata matching the Google Workflows execution. If there is no match, the Workflows will wait a short period and retry this function again. Once the matching CSV file is found, the scan is complete, and the function returns the vulnerability data and the CSV URL to download the file.

5. Other resources

There are some supporting resources used in this solution, which are described below:

• Google Cloud Storage Bucket: This bucket is used to store artifacts and scan results. The artifacts are things such as the Cloud Functions build files, startup scripts, scan results, etc.
• Google Cloud Networking: These are the networking resources required to have the solution running on GCP, such as a virtual network, subnet, firewall rules, Cloud NAT, etc.

6. Response payload

The response payload is returned by the Google Workflows to the caller, which in this case is the Golden Image Pipeline. The response contains the number of vulnerabilities by severity as well as the URL to download the scan report from Cloud Storage. An example response payload is shown below:

7. Sample Trivy CSV report

Below is a snippet of the Trivy CSV scan report:

Conclusion

In this article, we showed a way to integrate two different vulnerability scanning solutions into your Golden Image Pipeline using GCP Workflows and GCP Cloud Functions. We showed two different approaches to image vulnerability scanning, one being an open-source solution (Trivy), and the other being a commercial-off-the-shelf solution (Tenable.io). Either of these solutions can accommodate most common use cases. These vulnerability scans can be a useful step in your image process to fail builds that meet a certain threshold of vulnerabilities. This is a great way to automate the vulnerability management process for images within your environment. You can then limit your instance deployments to only allow the use of approved Golden Images.

Integrating vulnerability scanning into your Golden Image Pipeline in GCP was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Katterine Rodriguez and Victor Souza — Specialists, Build by McKinsey

We recently had the pleasure of participating in QCon, a global conference that gathers the best engineers from top-notch innovation companies. The event covers a wide range of relevant software engineering and leadership topics related to the kind of projects that have been carried out in our region.

We have learned a lot about how we can drive innovation and create meaningful impact for our clients by networking with people that face similar challenges and are implementing the best practices in technical projects. One of the most meaningful sessions was “Navigate complex environments and evolving relationships” by Jennifer Davis, Engineering Manager at Google, who shared valuable information about how to build a team and align it to deliver business value.

Navigating complex environments and evolving relationships

It’s no secret that companies expect staff to collaborate effectively, communicate well, and focus on their missions from day one. However, in the real world, such things do not always happen without the intentional effort to create an environment that fosters autonomy, trust, diversity of opinions and recurrent learning & improvement. The first step of this intentional effort is the acknowledgment that all team members must learn and practice how to collaborate effectively. This is known as team development.

The researcher and psychologist Bruce Tuckman developed a model that described the stages through which a group of people turn into a high performing team. His model has five stages — forming, storming, norming performing, and adjourning — and it is used as a reference by agilists.

• Forming stage: This happens when the team first comes together. Most members are kind and motivated, but some of them might be anxious and they look to a group leader for direction and guidance on how to tackle problems.
• Storming stage: This begins to occur while planning and executing tasks. At this stage, team members express their own points of view and showcase their personal work styles. In addition to that, it may raise structural issues, power struggles, confusion, and unclear role definitions.
• Norming stage: At this point, team members start to create stronger relationships, cohesion, demonstrate a sense of trust, and resolve their differences. The teamwork achieves better capacity to solve problems and at the same time, the leader starts to share responsibility with the group.
• Performing stage: When teammates focus and share the same mission and vision, exchange feedback with project managers and offer positive support to each other, team leaders can delegate more tasks and give autonomy to their team players.
• Adjourning stage: This happens once the job is done and it’s time for team members to leave the project. There are often feelings of sadness and reflection. It’s the perfect opportunity to recognize team efforts and achievements.

Understanding Tuckman’s group development process can help you enhance the performance of your teams. The question is, how can you help your team to go through those stages in the most effective way? According to Jennifer there are three key areas that drive team development:

1. Functional Leadership

“Leadership is not defined by the exercise of power but by the capacity to increase the sense of power among those led. The most essential work of the leader is to create more leaders.” — Mary Parker Follett, The Creative Experience, 1924

Awareness & Trust

It’s crucial that people feel respected for who they are, what they know and what they do. You should therefore get to know your team by learning about everyone’s motivations, objectives & concerns in order to recognize the strengths and weaknesses of each team member.

Another element is ensuring you share your own core values, experiences, and vulnerabilities. This will allow the team to get to know you better and you can start building trust with one another.

Open communication

Several management tools help us to organize and formalize tasks, but that is not enough to break the ice among teammates. A lack of open communication increases the likelihood of difficult tasks failing due to team members not sharing estimations, details or scenarios and avoiding suggesting improvements or reporting risks on time.

To facilitate open communication, offer teammates face-to-face virtual or physical spaces and increase probabilities for unplanned interactions among them. In addition, provide spaces where the team and business stakeholders can share a common vision of the final product.

Team Norms

Team Norms are a basic set of agreements defined by the members of the team about how to work well together. The teammates should discuss common practices and processes and define agreed upon behaviors.

For example, the team norms could include:

• Working inside scheduled sessions
• Establishing clear goals for each work session
• Focusing on resolving problems without placing blame
• Raising the alarm if there is a potential risk
• Keeping an open mind for colleague suggestions
• Maximizing the reuse of previous workstreams

Delegate leadership

Functional leadership lies in the assumption that all individuals are capable of leading. For the team to scale, leadership should be developed to suit the pace of each team member while making the team leader’s intentions clear so that the people have the right opportunity and support required to learn over time.

This doesn’t mean that clearly defined roles and responsibilities aren’t in place. Instead, this allows all team members to hold different leadership roles and fosters ownership of different parts of the delivery. It also facilitates knowledge sharing, workload and even work life balance to be spread more fairly across the team.

2. Boundary setting

Team rituals

In all dimensions of life, people have rituals, such as enjoying a cup of coffee to start the day with your coworkers or eating pizza each Friday to spend time with your family.

A ritual can create a strong, long-lasting personal connection so it’s a powerful tool to boost your team’s performance. If you promote some rituals in daily routines, you can help to promote an overarching team identity. A practical example is sharing a common playlist which all team members feed into that marks the start of each work session. Similarly, you could use an online wheel in order to randomly assign the order in which each team member participates during the daily ceremony session.

And at McKinsey, we also have wonderful team rituals. In our projects, it’s common to take a five-minute personal KPI (Key Performance Indicator) session before a daily check-in to keep track of everyone’s personal development points like “do exercises three times a week” or “only drink zero sugar soft drinks”. Another good example is to schedule a dinner from time to time to reinforce connections between the team members. And with that, everyone can motivate their colleagues to accomplish their personal KPIs and create strong relationships.

It’s important to select and define the team rituals with everyone so they can suggest what they want to include, engage, and enjoy each session. This isn’t something that should be dictated by the team leader, instead it should become part of the team’s ways of working.

Set and uphold boundaries

Setting boundaries allows us to get control over our family, friendships, projects, work life and how we effectively split our time and effort. If we are talking about our work, it’s important to establish boundaries ahead of team collaboration.

Team members should agree on the boundaries they need to adhere to so everyone feels comfortable working together. So, identify roles and responsibilities for each member of the team, the amount of time they should spend at work, their main priorities, when and how they communicate with each other, what the official channels of communications are, and determine capacity and speed of each member of the team. Offer alternatives to find win-win solutions and maintain a friendly work environment while supporting each other’s boundaries.

Here’s an example Jennifer shared on how she documents boundaries and set expectations with her team:

3. Learning and Adaptability

Individual feedback sessions (1:1)

This is a session between two members of the team where they both can look back on previous stages of the project, to evaluate and share their thoughts about the progress made. These feedback sessions allow people to reflect, encourage, recognize, learn, and seek ways to help their peers in their personal and professional growth. It’s also vitally important that these sessions are free of judgement so team members can express opinions openly, to strengthen communication, trust and remove the fear of making mistakes.

Learning opportunities

Change is constant, so it’s important to prepare you and your team to adapt and overcome challenges as they emerge. Jennifer believes that each team should dedicate time to training and learning during their planning sessions.

Discuss this with your team as well as team leaders. Your team’s ability to tackle problems depends on their knowledge and motivation, so having both training and learning sessions is essential to foster innovation, create business differentiators and should be seen as an investment.

Attending conferences can be a great way to keep your team updated on current tech trends and emerging technologies but also taking the time to attend technical leadership training can help develop your team members’ skills.

Implement Decision Records

This is a technique that has been increasingly adopted over the last few years, whereby using simple text-based documents describing each technical decision allowed users and other engineers to understand the rationale behind any decision that shaped the product or software.

These documents should consider things like:

• Purpose, definition, and consequences of this technical decision
• Urgency of this decision
• Who is making this decision — an individual or a group?
• How is this decision going to be enforced?
• Who needs to be made aware of and align with this decision?

Jennifer also shared a link to her team’s decisions records here (https://bit.ly/emblem-decisions) and we’d like to also share with you the Architecture Decisions Record (ADR) GitHub repository (https://adr.github.io/) where you can find more details about this technique and find other examples and templates to use with your team.

To sum up, Jennifer explained how difficult it is to navigate complex environments and evolving relationships in development teams, while also giving us a set of tools to build strong team relationships through open communication, and incredible teamwork.

We believe that being able to foster a trustful, knowledgeable, and balanced environment is essential for a healthy work environment. It is vital to identify what stage our team is at, and the key areas that drive its development to propose appropriate strategies that support rapid evolution to ensure high performance. Remember that a team’s culture and ability to operate properly takes time. Include people at all levels when making an action plan to help create healthy and efficient teams.

Overheard at QCon 2022: Navigate complex environment and evolving relationships was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

By: Tassio Abreu — Digital Specialist, McKinsey & Company

Anyone that has ever attended QCon will no doubt agree that it truly is one of the highlights of the year for software engineers and 2022 was no different. Microservices have always been an area of interest for me and one our clients regularly want to discuss with us. I made sure Microservices was a key focus of my time at QCon.

As you’d have guessed from the title, this article shares my key takeaways from the conference on how I recommend to design and deliver effective microservices. However, before doing so, it’s important to understand the benefits of microservices and how to secure the most value from the approach.

Starting your microservices journey

Everything starts with design. When new teams or new products are being conceived, many jump straight to microservices as this style of architecture is very much on trend currently. It’s important to differentiate between architectures you’re interested in and those that you really need, though.

During his QCon session, Chris Richardson, software architect and serial entrepreneur who founded Microservices.io, used metaphorical dark energy and dark matter to compose a memorable framework to help with exactly this, focused on repulsion forces and attraction forces.

He defined the repulsion forces, in this context, as:

• Simple components: when services are segregated, it’s much easier to analyze a codebase from scratch and understand its entirety
• Increase team autonomy: teams can focus on small pieces or business functions and thus be more responsible for its end-to-end success, from implementation to customer impact
• Fast deployment pipeline: smaller systems are deployed faster, the lead time for DORA metrics is often faster and from code to deployment benchmarks tend to be closer to elite levels
• Support multiple technology stacks: you can always use the best technology at hand, whether to increase performance, to use a framework that is more advanced and can lead to improved ergonomics and engineering time, or for compliance purposes
• Cost-effective scaling: you can scale pieces of your application separately, allowing for a new concept of scaling through container density, which in turn diminishes the chances of having to overprovision up to known peak capacity to avoid problems in uptime
• Segregate highly available components: you can allocate different services in different infrastructures, and this can cause variations in both cost and availability

On the other hand, he defined attraction forces as:

• Simple interactions: importing a class or file and using it directly will always be simpler, cheaper and fast than having to connect through a network with another component
• Prefer ACID over BASE: avoiding eventual consistency allows for better and an always-true system behavior — there’s no risk that information being hot-read is inconsistent
• Minimize runtime coupling: service availability will depend less on other moving pieces. The more distributed a system is, the more expensive it is to implement resilience, as more moving parts often result in increased complexity and a higher risk of failure
• Efficient inter-service communication: communication often happens in-memory, ensuring it’s fast and predictable
• Minimize design time coupling: your service will need to change less frequently, especially in response to external forces. Imagine having to redeploy your system regularly due to a dependence on an external REST API of a service that keeps changing. It’s far from the ideal approach

As a rule of thumb, it’s recommended to begin with attraction forces, as keeping them together is often easier and provides engineering teams with a stronger platform for eventual success. It’s also important to keep in mind that you don’t need a distributed team at this stage, unless it’s project critical.

After starting small, keep challenging your architecture with real, proven hypotheses for repulsion. It’s here that distributed systems begin to offer value, as although they’re more complex to maintain, they have better end-game potential for scalability and resilience. This does come at a price, but Chris defined and shared multiple patterns to help here.

It’s important to understand that system design is an iterative process, so constantly reevaluate how to evolve your system. It can even prove beneficial to decompose it into multiple sections in terms of decoupling, scalability, enhanced team topology and enabling two-speed IT. The latter is a framework by McKinsey which showcases how teams that require stability while delivering disruptive, innovative solutions.

So how should you iterate on your system design? The following is derived from Chris’ pattern language:

• Define the context of your system, e.g: an eCommerce system being evolved to increase conversation rates
• Enumerate the problems being tackled, e.g: there’s a big churn before checkout due to lengthy user log-in times
• Note the forces that help prioritize each problem, e.g: new team members must quickly become productive as the checkout service is critical and new deployments can’t happen regularly due to the risk associated
• Design the solution, e.g: the checkout functionality will be broken down into a new microservice, to decouple its lifecycle from other components that can and should change more often to adapt to users’ behavior
• Implementation details, code commits and effective code changes that bring your new system to life
• Resulting content, e.g,: C4 diagrams, new ADRs and other content to illustrate how the system will behave and interact

Taking care of overcomplicated architecture early

At the second session I attended at QCon, Cassandra Shum, enterprise architect and former member of the Technical Advisory Board at Thoughtworks, discussed the importance of avoiding overcomplicated architecture which perfectly complemented Chris’ session. She strongly believes that driving a culture of engineering excellence takes care of net-zeroing tech debt early in any project, and I have to agree. Especially having now witnessed her framework which follows-up on the growing complexity of architecture based on symptoms vs remedies.

The following are the areas suggested engineers focus on to diagnose overcomplicated architectures:

• Distributed monoliths: this is the worst possible scenario. Teams try to dive into microservices, but end up with replicated code, dysfunctional patterns and outputs which don’t reap any benefits of either architecture
• Tracing, logging, monitoring and visibility without benefit: nobody knows what’s going on and logs become an expensive clone of the famous green screen in The Matrix which rarely leads to problem solving
• Too much abstraction: indirection can lead to delays and unnecessary moving parts that bring additional risk and needless complexity
• High cognitive load: the system is so complex to understand that onboarding new developers, irrespective of how good they are, is always a burden. Changing a single line of code has so many potential side effects that teams simply prefer to stagnate
• Difficulty making technical decisions: too many cooks spoil the broth, and this proves it. When too many people need to be brought into discussions prior to making changes, teams can’t evolve their business functions
• A ‘build in-house’ mindset: The perfect example is a fintech implementing its own authentication system instead of focusing on innovation that will set them apart from the competition. Every time a business builds something in-house which could have been bought “off-the-shelf”, that’s one less team potentially steering the company forward

Remedies on the other hand do exactly what they say on the tin, providing solutions to the aforementioned symptoms:

• Have clear domain boundaries: align your business with your code. This can be achieved by having a good, yet lightweight, understanding of domain driven design, as well as spending time defining an ubiquitous language that is spoken from directors down to the engineers seamlessly
• Observability and monitoring: find a process which allows real-time inspection of the inner workings of your system, from the outside. Don’t ignore anomalies, be alarmed by them. Ensure people around you understand the system and the importance of cooperating with product teams on debugging
• Lightweight architecture decision records (ADRs): There shouldn’t be excessive or unknown side effects, so changing the system shouldn’t be a daunting task
• Cross-functional requirements: leverage patterns like microservices chassis to reduce the time teams spend work on activity which could be done just the once before being centralized and reused. The likes of sidecars, proprietary libraries and packages really help at this point
• Remove the hero culture: if critical knowledge is owned by one person, you have a very low truck-factor as if that person is unavailable or leaves, a project can come to a quick halt
• Controlled technical debt: your team should know how to measure technical debt, be aware of issues with code quality and react to such issues to ensure best practice
• Developer effectiveness: codes can be changed by a new engineer without risk as it’s protected by automation and has controlled and known side effects
• Platform thinking: product teams are incentivized to contribute, donate and eventually platformize team-specific tech assets to the company as a whole
• Commodity vs differentiation: teams should spend as much time as possible working on new features and potential revenue streams or end-user perceived quality, instead of unnecessary heavy lifting against the system’s own weight and inertia

Effectively operating your distributed system

With your microservices journey now well under way and a clear framework in mind for tackling overly complex architecture, attention should turn to effectively operating your distributed system. During his talk at QCon, Wanderlei Souza– a distinguished engineer from Build by McKinsey — put together a collection of best practices that feels like the missing piece of the designing and delivering microservices puzzle.

However, before introducing a distributed system, there are three crucial steps that should be taken:

• Have design patterns as a handy problem-solving resource: common solutions to common problems are your best bet to keep your system manageable and evolvable over time. Make sure engineers are fluent on a pattern language for distributed architectures
• Create a culture of parsimony: focus on simple solutions. Use a framework capable of taming overengineered architectures to avoid unnecessary — and usually accidental — complexity
• Implement observability and alarming: you can use a platform that better suits your team preferences, needs and budgets

Once these steps have been taken, it’s vital that engineering teams work together with the wider business stakeholders to align on objectives and outcomes. With that in mind, the following considerations will ensure your distributed system is infinitely more effective:

• Don’t forget about the humans: make sure that engineering teams and product / business focused colleagues are aligned. Avoid the scenarios in which a new Black Friday push notification campaign can generate a traffic surge that damages or even causes service downtime.
• Master patterns and trade-offs for service communication: favor orchestration for complex workflows, choreography for scalability and weaker coupling
• Design for resilience: consider concepts such as:
• Graceful degradation
• Fault tolerance
• Circuit breakers
• Bulkhead
• Business-aware health checking
• Distributed tracing for end-to-end debugging
• Log aggregation
• Anomalies on application performance metrics
• Disliking TDD but resisting letting go of high testability: make sure you can easily simulate real-world situations at test environments, record the operations of a given day and simulate or replay these deliberately. It’s also key that you’re notified of changes on external integration contracts and that you’re able to reproduce composed scenarios when lag or system failure occurs
• Measuring your success: it’s vital you can evaluate the overall quality of your system against a well-respected industry standard, such as ISO 25010

Microservices aren’t a silver bullet

Operating microservices isn’t a walk in the park and certainly isn’t the silver bullet for achieving engineering excellence that many paint it as. Despite the acknowledged complexity, we all accept that being aware and understanding these concepts can help any kind of software engineer or architect in any sector — whether that’s a cloud-native, written-from-scratch startup, or an incumbent company that needs to be brought into the 21st century.

Eight years after Chris’ ground-breaking first talk about the microservices architecture, tools and patterns, community knowledge on the topic has grown exponentially. It’s interesting to see how widespread and well-adopted the architecture is and how its applicability suits the need of modern businesses in the era of agile, team topologies and subscription services. Especially as, when managed well, microservices can be a lever for increased engineering agility.

So while it’s a complex area of focus, if your team relies on patterns with a proven track record, allows for observability and takes the time to implement controllability early in the microservices journey, success will be within touching distance.

Overheard at QCon: Designing and delivering effective microservices was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

For those who have never experienced the wonders of QCon, it’s one of the leading events in the software engineering calendar, focusing on the latest software trends and cutting-edge innovations. Taking place in San Francisco late last month, I was fortunate enough to attend and witness first-hand discussions on the future of the software development industry and certainly had my eyes opened by some of the talks I attended.

As the title of this article suggests, I’ll be focusing on two areas of inspiration following a couple of insightful sessions I attended during my time at the conference.

Accelerating teams and scaling-up engineers

One such session I attended was hosted by Argentinian eCommerce business Mercado Libre which describes itself as a “marriage between Amazon and PayPal.” The discussion focused on scaling-up engineering teams, with the key motto hammered home being ‘accelerating teams, controlling organizational entropy.’

To say the business has been on quite a journey since launching its architecture as a monolith in 2000 would be an understatement given that today it has the Fury platform to enhance developer experience and have been scaling the number of engineering teams since 2020. In fact, the average day sees Mercado Libre responsible for 10,000 deploys, 26,000 microservices and training 700 machine learning models.

This journey was a key discussion point during the session, outlining that 83% of Mercado Libre’s developers have less than two years with the organization. When it had a more modest team, it could get away with providing a single career path which saw seniority aligned directly with managerial responsibilities. However, as Mercado increased its headcount, it also created a parallel career path where employees could grow their technical skillset without managing other developers. This is built on the premise that experts lead experts, similar to the approach Harvard Business Review suggests is a primary reason for Apple’s success.

The team structures saw 85% of developers positioned within delivery teams responsible for building customer-facing products, while the remaining 15% were part of platform teams. These deliver teams are cross-functional and consist of:

• Front-end developers
• Back-end developers
• Machine learning engineers
• Native developers
• Data engineers
• User experience experts

Simultaneously, Mercado Libre’s platform teams are split by:

• Cloud engineers
• Security engineers
• Machine learning engineers
• Quality engineers
• Data analytics engineers
• Site reliability engineers

The positioning of product managers is particularly interesting, as Mercado Libre doesn’t see the need to have a product manager per team. Instead, it prefers all members of its delivery teams to be “product managers.” The way Mercado approaches this makes sense. They have a UX designer in every team, who coach teams to see customer perspectives. On average, this means it has one product manager for every 25–30 engineers, a structure that has worked wonderfully for them during their growth story. This structure makes sense for them, because they have a tightly aligned UX team and also ensure that they have a UX person on every team.

Its management team also strived to ensure that 90% of decisions were made by those closest to the work, with only 10% of decisions escalated to leadership and 1% of those positioned as “golden nuggets”. These are strategic pieces of decision-making that executives look at very seriously and support the direction of travel for engineering teams.

Alongside the team structure, Mercado Libre also has a robust system which helps to accelerate the team, facilitate scaling-up and drives innovation. It has a robust platform that takes care of risk scoring, machine learning and cloud computing while reducing the cognitive load on product engineering teams.

Mercado Libre’s Fury platform does exactly that, being available to all its developers, who could instantiate it with a few commands. The platform prides itself on providing a consistent developer experience, removing complexity, and offering a cheap service that allows product engineers to innovate at pace. It does so by taking care of the following behind the scenes:

• Software management through GitHub, Jenkins and Docker
• Compute services through GCP and AWS
• Data services through GCP, AWS and Oracle Cloud
• Service management through New Relic, Opsgenie, Jupyter and Bugsnag

One key to the Fury platform’s success outlined during the QCon session was that it followed Mercado Libre’s transition to a microservices-based architecture, meaning it invested significant time and effort to decouple its architecture before building the developer platform.

We’ve seen a trend towards clients attempting to build ‘platforms’ without paying the required attention to the coupling of their architecture. Unfortunately, it’s a recipe for failure which takes serious work to undo. Instead, organizations should approach decoupling by understanding the domains and bounded contexts within the system architecture.

Fury is a perfect case study of how it should be done, with a modern technology firm considering leadership, teaming, talent and architecture and understanding each are key to delivering success. It will be interesting to see if more businesses take such an approach having witnessed the session.

Is continuous deployment beneficial?

Another hugely popular session I attended during my time at QCon this year was a fantastic talk by representatives at Lyft on its journey to continuous deployment and the struggles it faced on the way to ensure developers could deploy code effortlessly within minutes.

At this stage it’s of course important to differentiate between continuous deployment and continuous delivery. On this, Jez Humble, author of the seminal work, Continuous Delivery, said: “…when we can release on demand at the push of a button, during normal business hours, we are doing continuous delivery.” Simply put then, continuous delivery and continuous deployment attempt to automate the entire deployment process to production. Jez also believes that “the key thing we should care about is not the form, but the outcomes: deployments should be low-risk, push-button events we can perform on demand.”

Many organizations would envy Lyft’s starting point — its developers could deploy from staging to canary to production in half a day. It’s commendable that the business has continued improving, as it outlined the following four pain points:

1. Deploying took half a developer’s day
2. Many commits deployed at once
3. Commits from many different authors, each with various changing components
4. Many uncoordinated changes often made it difficult to figure out what was happening during incidents

Lyft started tackling this by creating a Deploys Team, solely accountable for reaching continuous deployment to reclaim developers’ time. The key here was empowering a multi-disciplinary team to own the entire thought process of deployment. This aligns perfectly with our advice to clients on how to motive talented engineers, by providing autonomy and a clear sense of purpose.

Alongside the need to reclaim its developers’ time, Lyft supplemented the Deploys Team objective with three design tenants:

1. Automated — remove as much manual intervention as possible
2. Scalable — allow teams to add integration checks and alerts to govern deployments efficiently
3. Responsive — the system should capture the mental context that developers have now

This approach provided Lyft’s developers strong foundations to build on, while ensuring teams had smaller deploy sizes was a key metric. It was mentioned that Lyft needed to measure quicker time to production since roadmaps and feature complexities varied by teams. This is one of the infamous DORA metrics we strongly advise most clients to keep an eye on. However, Lyft started from a different launch pad so could afford not to monitor its lead time to deploy that closely.

It was also revealed that Lyft’s Deploy Team built its deploy system in-house, using the following tech stack:

1. Backend was in Go, the frontend was in React.js
2. PostgreSQL — for both online storage and analytics queries
3. ReST APIs

• DeployAPI tracked the deployed state
• DeployView interacted with the state
• AutoDeployer continuously deploys when safe

When questioned on this approach, Lyft stated building a tool in-house worked well for them as it allowed full customization. It also highlights the level of empowerment offered to the Deploys Team and proves a team with autonomy and a sense of purpose can produce incredible results.

The following is a visual representation of Lyft’s fundamental data model of its deployment system — jobs.

A few jobs came together to form a pipeline.

Each pipeline would go through several states.

Lyft has no rollbacks built into the model and no rollback state in its state machine. A developer could trigger a deployment with an earlier build and it avoided concurrency issues. If Lyft has to rollback, it would have to pause the pipeline.

Lyft also revealed a sample checklist which helps them determine completion for a pipeline to pass from a waiting state to a running state:

• Has the revision passed continuous integration tests?
• Have business metrics been affected?
• Have integration tests passed?
• Has bake time passed?
• Ensure no service alerts are firing.
• Would this cause an unintended rollback?
• Is the owning team present and working?
• Is the job more than 14 days old?

This is an excellent checklist for moving from one state to another, although if considering a similar approach I recommend adding as many non-functional requirements as you need to the tests, along with code linting and some coding standards.

Lyft closed the session by outlining key lessons it learned about continuous deployment along its journey:

• A less delicate rollout would’ve been fine. Its developers were excited by the prospect of continuous delivery and Lyft was working to make its developers’ lives easier
• Not having automated rollbacks worked well for them
• Previously, it was easy for ineffective changes to ship to production — continuous deployment has significantly reduced such errors
• Continuous deployment. It has made developers’ lives easier and given them more confidence to make changes across projects
• Continuous deployment reduced Lyft’s exposure to CVEs. A CVE announcement used to result in a scramble; now, they use the continuous deployment pipeline with little manual work

At McKinsey, we regularly tell clients that continuous deployment is a time-intensive process to introduce but is worth the investment. The journey lasted a few years but has resulted in empowered engineers with the latitude and space to innovate at pace. With backing from senior leadership teams, it pays off in the long run.

They are my key takeaways from just two brilliant sessions I attended at QCon 2022 on behalf of McKinsey. My colleagues will be sharing their own insight from other QCon talks in the coming weeks, so watch this space.

Overheard at Qcon: Accelerating teams and continuous deployment was originally published in McKinsey Digital Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.