There are plenty of tutorials out there on how to get started building MCP servers and clients, but most of them stop after building functional requirements and do not describe the authorization and authentication implementation.

Some tools do not need to authenticate users, because they only provide read access to publicly available resources. But the advanced use cases will need access with an authenticated identity.

This article will describe:

• history of authorization in MCP
• how EntraID can be used to secure an MCP server (and what features are missing)
• how to build a Typescript MCP server secured by EntraID which can be used in VSCode with Github Copilot
• provide users tools, depending on which permissions they have

The demo code is provided in the repository at https://github.com/patst/entraid-mcp-server .
Clone the repository and follow the instructions below to configure the required variables.

Model Context Protocol

The Model Context Protocol is a standardized communication protocol that enables AI agents to interact with external tools, databases, and services. It can provide the bridge between large language models and the already existing APIs (enterprise) applications.

MCP servers expose ‘tools’ that AI agents can discover and call. These tools can range from simple utilities like text processing to complex operations like database queries, API calls or file system operations. The protocol defines how these tools are described, discovered, and invoked, creating a consistent interface for AI agent interactions.

More about the MCP basic architecture and building blocks can be found in the awesome MCP specification at https://modelcontextprotocol.io/specification/2025-06-18/architecture

Authentication and authorization in MCP

The MCP protocol’s first specification was released in November 2024.
Since then, the security topic has received a lot of attention in the specification releases:

1. 2024–11–05: Only mentions that “MCP cannot enforce security principals” and implementation should “follow best practices”
2. 2025–03–26: Dedicated Authorization chapter. OAuth 2.1 defined as security standard. MCP Server acts as Authorization Server
3. 2025–06–18: The MCP server now only takes the OAuth resource server role. A protected resource metadata document (see below) needs to be provided

It is worth reading through the specification; great to see the improvements.
Especially the last specification release, which treats the MCP server as a resource server in an OAuth sense, really simplifies the implementation steps required.

The authorization flow is described here:

EntraID configuration

EntraID is used as an Identity Provider in a lot of companies.
It supports OAuth as an authentication protocol with public and secret clients.

For this tutorial, we will need an Application Registration which can be created using the Azure CLI:

az ad app create --display-name "MCP-Server Demo" \
  --requested-access-token-version "2" \
  --sign-in-audience "AzureADMyOrg"

Open the application registration in the Azure Portal for the final configuration to expose an API:

• at `Expose an API`, expose a scope named `user_impersonation` for the app registration (it may already exist, because the Azure CLI created it)
• at `Expose an API`, add the ClientIds as preauthorized client applications:
  - `aebc6443–996d-45c2–90f0–388ff96faa56` — default Client ID for VSCode. This is required to use the MCP Server with Github Copilot from VSCode
  - `04b07795–8ddb-461a-bbee-02f9e1bf7b46` — Default Client ID for Azure CLI, allows fetching of tokens for test purposes

We need the ClientId of the created application registration together with your TenantId.
Create a .env file in the root directory of the demo repository with this content:

AZURE_TENANT_ID=<valueFromEntraID>
AZURE_CLIENT_ID=<valueFromEntraID>
BASE_URL=http://localhost:3001

OAuth resource server implementation

The MCP protocol requires the server to provide a OAuth 2.0 Protected Resource Metadata (RFC9728) endpoint.

The demo project uses expressjs as web server and the code for the endpoint looks like this:

app.get('/.well-known/oauth-protected-resource', async (_: Request, res: Response) => {
  res.json({
    resource: `${process.env.BASE_URL}/mcp`,
    authorization_servers: [`https://login.microsoftonline.com/${process.env.AZURE_TENANT_ID}/v2.0`],
    bearer_methods_supported: ["header"],
    scopes_supported: [`api://${process.env.AZURE_CLIENT_ID}/user_impersonation`]
  });
});

Whenever an MCP Client calls the MCP server and the server decides that the client needs to be authorized, the server returns the metadata which points the client to an identity server and provides all required information to fetch an access token.

We provide a list of supported scopes, which contains only our user_impersonation scope at the moment.
We use the environment variables for the EntraId, TenantId and ClientId we configured above.

Authentication middleware

Our server should support both authenticated and anonymous access.

The idea behind that is to allow clients to express they want to use the server in anonymous mode. The server then only provides the tools which can be used without an identity.
The client must specify the HTTP header `x-use-anonymous` to state that intent.

const authMiddleware = async function (req: Request, res: Response, next: any) {
  if (req.headers["x-use-anonymous"]) {
    // Skip authentication for anonymous mode
    next();
  } else {
    // Require Bearer token authentication
    bearerAuthMiddleware(req, res, next);
  }
};

The `bearerAuthMiddleware` uses a implementation provided by the MCP SDK:

import { requireBearerAuth } from "@modelcontextprotocol/sdk/server/auth/middleware/bearerAuth.js";

//...

// this middleware checks authorization token and returns a WWW-Authenticate error with infos where to get an auth token
const bearerAuthMiddleware = requireBearerAuth({
  verifier: tokenVerifier,
  requiredScopes: ["user_impersonation"],
  resourceMetadataUrl: `${process.env.BASE_URL}/.well-known/oauth-protected-resource`
})

I was surprised to not find a predefined `tokenVerifier` implementation.
Either it is missing in the SDK or I was not able to find it.
The token verification is very important to check if the access tokens are really issued FOR your application and signed by your EntraID tenant.

The JSON Web Key Sets URL provided by EntraId is used for checking the signature of the token:

const jwksClient = new JwksClient({
  jwksUri: `https://login.microsoftonline.com/${process.env.AZURE_TENANT_ID}/discovery/v2.0/keys`
});

const tokenVerifier: OAuthTokenVerifier = {
  verifyAccessToken: async (token: string): Promise<AuthInfo> => {
    // validate signature
    const tokenHeader: string = token.split(".")[0] as string;
    const signingKey = await jwksClient.getSigningKey(JSON.parse(atob(tokenHeader))["kid"]);
    await new Promise((resolve: any, reject) =>
        jwt.verify(token, signingKey.getPublicKey(), (err, decoded) => err ? reject(err.message) : resolve(decoded))
    );
    console.log("Access token verified successfully");
    
    // Extract relevant claims
    const tokenPayload: string = token.split(".")[1] as string;
    const tokenParsed: any = JSON.parse(atob(tokenPayload));
    // Convert the response to AuthInfo format
    return {
      token,
      clientId: tokenParsed.aud,
      scopes: tokenParsed.scp ? tokenParsed.scp.split(' ') : [],
      expiresAt: tokenParsed.exp,
      extra: {
        groups: tokenParsed.groups ? tokenParsed.groups : [],
        username: tokenParsed.preferred_username,
      }
    };
  }
}

The `return` part of the function returns an `AuthInfo` object defined by the MCP SDK. The `extra` attributes allow specifying arbitrary values, which can later be used for making authorization decisions if the scopes are not enough.

Dynamic tool access based on authentication

Now we are able to authenticate users. The next step is to make authorization decisions based on the identity.

There are two possible places for that:

1. provide different lists of tools, resources, prompts, etc. based on an identity (e.g. hide admin tools if the user has no admin permission)
2. Return different output during tool calls (e.g. a user can only look up customer information he has access to)

Different tools based on user identity

First, we decide to show a tool only if the user is authenticated.
This is done by accessing the AuthInfo object during the ListToolsRequestSchema API call.

server.setRequestHandler(ListToolsRequestSchema, async function (_, extra) {
  const authInfo = extra.authInfo;

  // Base tools available to everyone (including anonymous users)
  const tools: Tool[] = [
    {
      name: ToolName.ECHO,
      description: "Echoes back the input",
      inputSchema: zodToJsonSchema(EchoSchema) as ToolInput,
    },
  ];
  
  // Additional tools only for authenticated users
  if (authInfo && authInfo?.extra?.username) {
    tools.push({
      name: ToolName.USER_DETAILS,
      description: "Returns details about the authenticated user and their permissions.",
      inputSchema: zodToJsonSchema(UserDetailsSchema) as ToolInput,
    });
  }

  return { tools };
});

In that example, the `user details` tool is only displayed if the user is authenticated. We could read the `groups` attribute from the `authInfo?.extra?` to make decisions based on groups as well.

Different response for tool call base on user identity

The user’s identity is available during the `CallToolRequestSchema` as well and can be used to display different data, depending on user permissions:

server.setRequestHandler(CallToolRequestSchema, async (request, extra) => {
  const { name, arguments: args } = request.params;

  switch (name) {
    // ...
    case ToolName.USER_DETAILS: {
      console.log("Returning user info", extra.authInfo?.extra);
      return {
        content: [{ type: "text", text: `Detected user ${extra.authInfo?.extra?.username}` }],
      }
    }
    // ...
  }
});

Another interesting use case is to use the on-behalf-of flow to call other APIs on behalf of the currently logged-in user. Details for that can be found here: https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow

This would allow passing the user’s identity down the call chain for transitive APIs. (This only works for real users and not service principals)

Running the demo

After finishing the implementation, we now want to test the MCP server.

Start the MCP server with running `npm run dev`.

Testing with MCP inspector

We will use the MCP Inspector for local testing.

Run `npm run inspector` in the root folder to start the inspector.

Unfortunately, EntraID is not fully compliant with the MCP authorization specification:

1. the Entra OIDC discovery document is missing the code_challenge_methods_supported attribute (only workaround: serve a copy of the document)
2. resource request parameter is not supported (see AADSTS901002)

These two points do not allow using the OAuth flow provided by the MCP inspector. Instead, we will use the Azure CLI to get an access token and test our MCP Server (remember: we authorized the Azure CLI to fetch access tokens for our application in the EntraID configuration part):

az login # use your user to login
az account get-access-token --scope api://${AZURE_CLIENT_ID}/user_impersonation -o tsv --query "accessToken"

This will return an access token which can be copied in the Authorization part of the MCP inspector.

Integration test with Github Copilot in VSCode

Now the real test in VSCode together with Github Copilot.

Start the server (if not already running) with `npm run dev`.

Then open VSCode and configure your locally running MCP server in the `mcp.json`:

{
  "servers": {
    "local-mcp-server": {
      "url": "http://localhost:3001/mcp",
      "type": "http"
    }
  },
  "inputs": []
}

When VS Code connects to the server:

1. It discovers the authentication requirements from the `.well-known/oauth-protected-resource` endpoint
2. Prompts the user to authenticate with EntraID
3. Includes the access token in subsequent requests
4. Receives the appropriate set of tools based on the user’s authentication status

For testing anonymous access, simply add the `x-use-anonymous` header:

{
  "servers": {
    "local-mcp-server": {
      "url": "http://localhost:3001/mcp",
      "type": "http",
      "headers": {
        "x-use-anonymous": "true"
      }
    }
  }
}

Key takeaways

Implementing OAuth 2.0 authentication for MCP servers opens up new possibilities for enterprise integration while maintaining security best practices. This tutorial demonstrated how to:

• Configure EntraID as an identity provider for MCP servers
• Implement OAuth 2.0 Protected Resource Metadata endpoints
• Build authentication middleware that supports both authenticated and anonymous access
• Dynamically control tool availability based on user identity and permissions
• Integrate the secured MCP server with GitHub Copilot in VSCode

The MCP specification’s evolution from basic security mentions to full OAuth 2.0 support shows the protocol’s maturity and enterprise readiness. While some compatibility issues exist with EntraId (resource parameter support), workarounds allow for functional authentication flows.

With authenticated MCP servers, AI agents can now access enterprise resources securely, respecting user permissions and organizational boundaries. This enables more sophisticated use cases like personalized tool access, audit trails, and integration with existing identity management systems.

As the MCP ecosystem continues to grow, secure authentication will become increasingly important for enterprise adoption and real-world AI agent deployments.

Links

• Demo Repository: https://github.com/patst/entraid-mcp-server
• MCP Spec for Authorization: https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization
• Entra ID resource parameter not supported issue: https://github.com/microsoft/vscode/issues/254009
• code_challenge_not_supported issue: https://github.com/modelcontextprotocol/typescript-sdk/issues/832

Securing a Model Context Protocol Server with EntraID was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tracing is a very important component in a distributed system to get insights into what is happening and what might be slow or even broken.

Azure API-Management is a Platform-as-a-Service offering which provides an API-Management platform for APIs across different environments/cloud-providers. One of the provider components are API-Gateways which are responsible for routing, security, throttling, caching, and observability.

The different features can be configured and activated using Policies. Policies can be configured per API, for certain methods or globally for all APIs. Examples for policies are JWT validation, authentication to backend services or sending either synchronous or asynchronous calls to services.

For this article the observability of the API-Gateway is important (docs). Azure provides integration into the Azure ecosystem with connections to Azure Application Insights (metrics), Azure Monitor (logs/metrics) and EventHub (logging).

The piece unfortunately missing is the integration into a tracing solution.

This is where OpenTelemetry comes to the rescue. OpenTelemetry is the de-facto standard for observability, providing specifications and SDKs for several observability signals (e.g. Logs, Metrics, Traces).

With Grafana Tempo we already have a central place to store all our traces in. Our goal now is to be able to ingest traces from the API-Gateways to our central tracing backend.

Architecture

Most of the required components can be used from the OpenTelemetry project. The missing piece, is getting trace information from the API-Gateways to an OpenTelemetry collector.

A collector is a “Vendor-agnostic way to receive, process and export telemetry data” (source: OTel).

An OpenTelemetry collector has different parts which are described in this picture:

We can use the OTLP exporter to export traces to Grafana Tempo. To get the traces from the API-Gateway to the collector we will implement a custom OpenTelemetry receiver.

The receiver is responsible for getting trace information and then map it to the OpenTelemetry object model, which is described in the Tracing Signal specification.

Lucky for us there is already a well-written tutorial for Building a receiver in the OpenTelemetry docs.

We do not want to block API calls with our tracing solution, therefore we need an asynchronous communication way between the API-Gateway and the OpenTelemetry receiver. To implement that, we will use Azure EventHubs as event store.

The complete architecture diagram looks like this:

A request traverses the architecture in the steps described in the image above:

1. A User (or application) calls an API published in the API-Management. The request is send to the API-Gateway
2. A policy inside the API-Gateway checks if a traceparent header exists. If none already exists a new one will be created, otherwise only a new SpanId will be generated. The new traceparent header will be passed on to the API backend.
3. The requests is forwarded to the API backend contains the implementation
4. The API backend returns the response to the API-Gateway.
5. The required information for a trace is gathered and send as event to the EventHub
6. The caller receives the response for the API call (if there is one)
7. The custom OpenTelemetry receiver listens for new events in the EventHub and if there is one it will be converted according to the OpenTelemetry trace specification
8. The OpenTelemetry collector exports the trace to a trace backend, which can be Grafana Tempo or Jaeger.

Implementation

This is not intended as production-ready implementation, but as a starting point, if you want to build such an architecture.

First we need to bootstrap a OpenTelemetry collector project using the OpenTelemetry Collector Builder (ocb).

First we create a builder config and then generate the code from the configuration (builder-config.yaml ):

dist:
  name: otelcol-dev
  description: Basic OTel Collector distribution for Developers
  output_path: ./otelcol-dev

exporters:
  - gomod:
      go.opentelemetry.io/collector/exporter/debugexporter v0.115.0
  - gomod:
      go.opentelemetry.io/collector/exporter/otlphttpexporter v0.115.0

processors:
  - gomod:
      go.opentelemetry.io/collector/processor/batchprocessor v0.115.0
extensions:
  - gomod:
      github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.115.0

providers:
  - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.18.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.18.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/httpprovider v1.18.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/httpsprovider v1.18.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.18.0

# generate the code for the collector using the collector builder tool
# change the cpu architecture if you are not using linux_amd64
curl --proto '=https' --tlsv1.2 -fL -o ocb \
https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.116.0/ocb_0.116.0_linux_amd64
chmod +x ocb
./ocb --config builder-config.yaml

This way we can configure what extensions, receivers, exporter and processor should be included in our custom collector. Above is mainly the default configuration.

After that we have a skeleton generated which looks like this:

.
├── builder-config.yaml
├── ocb
└── otelcol-dev
    ├── components.go
    ├── components_test.go
    ├── go.mod
    ├── go.sum
    ├── main.go
    ├── main_others.go
    ├── main_windows.go
    └── otelcol-dev

After that we initialise our custom receiver code which follows exactly the OpenTelemetry tutorial.

mkdir apimtracer
cd apimtracer
go mod init github.com/patst/azure-apim-otel-receiver/apimtracer
cd ..
go work init
go work use otelcol-dev
go work use apimtracer

I do not want to go into the detailled implementation steps, but only highlight the required functions and structs. The whole example code can be found at https://github.com/patst/azure-apim-otel-receiver .

Configuration and event structure

First we need to define the possible configuration parameters for our custom receiver in the apimtracer/config.go file:

type Config struct {
 FullyQualifiedNamespace string `mapstructure:"fully_qualified_namespace"`
 EventHubName            string `mapstructure:"event_hub_name"`
 ConsumerGroup           string `mapstructure:"consumer_group"`
 StorageContainerUrl     string `mapstructure:"storage_container_url"`
}

These values can be used in our collector configuration later. It contains all required parameters for the receiver to connect to an Event Hub.

Next we define the event structure we are expecting to receive for each API request. This structure will later be mapped to a object adhering to the OpenTelemetry trace specification:

type ApimTracePayload struct {
    ApiName          string  `json:"api_name"`
    SubscriptionName string  `json:"subscription_name"`
    ProductName      string  `json:"product_name"`
    TraceParent      *string `json:"traceparent,omitempty"`
    ParentSpanId     *string `json:"parent_span_id,omitempty"`
    HttpTarget       string  `json:"http_target"`
    HttpMethod       string  `json:"http_method"`
    HttpHost         string  `json:"http_host"`
    HttpStatusCode   int     `json:"http_status_code"`
    HttpClientIp     string  `json:"http_client_ip"`
    RequestStartTime int64   `json:"request_start_time"`
    RequestEndTime   int64   `json:"request_end_time"`
}

We want to collect information about the duration, HTTP status codes, client ip, the API name and some more attributes which are later queryable. We expect a JSON string to be received from Azure Event Hubs which can be parsed to this struct (json.Unmarshall([]byte(eventData), &apimTracePayLoad)).

Mapping event to trace object

The complete mapping logic can be found in the trace-receiver.go file (source). Basically it extracts all information from the event and creates a OpenTelemetry span object:

 resourceSpan := traces.ResourceSpans().AppendEmpty()
 resource := resourceSpan.Resource()
 resourceAttrs := resource.Attributes()
 resourceAttrs.PutStr("apim.product.name", apimTracePayload.ProductName)
 resourceAttrs.PutStr(conventions.AttributeServiceName, "apim")
 resourceAttrs.PutStr("apim.api.name", apimTracePayload.ApiName)
 resourceAttrs.PutStr("apim.subscription.name", apimTracePayload.SubscriptionName)
 // ...
 apiSpan := scopeSpans.Spans().AppendEmpty()
 apiSpan.SetName(fmt.Sprintf("apim: %s %s %s", apimTracePayload.HttpMethod, apimTracePayload.ApiName, apimTracePayload.HttpTarget))apiSpan.SetKind(ptrace.SpanKindServer)
 // example traceparent: 00-a9b7d8bbcecf8c7d9d24c69cc86f694d-2defb8587ed6dfef-01
 // format: version-trace_id-parent_id-trace_flags
 fmt.Printf("Extracting span and trace id from traceparent %s", *apimTracePayload.TraceParent)
 traceString := (*apimTracePayload.TraceParent)[3:35]
 traceId, err := uuid.Parse(traceString)
 if err != nil {
  return nil, fmt.Errorf("error converting traceId to uuid: traceString: %s err: %v", traceString, err)
 } else {
  apiSpan.SetTraceID(pcommon.TraceID(traceId))
 }

 decodedSpanId, err := hex.DecodeString((*apimTracePayload.TraceParent)[36:52])
 apiSpan.SetSpanID(pcommon.SpanID(decodedSpanId))
 fmt.Printf("Extracted TraceId=%s and SpanId=%s, apiSpanId=%s", (*apimTracePayload.TraceParent)[3:35], (*apimTracePayload.TraceParent)[36:52], apiSpan.SpanID().String())
 if apimTracePayload.ParentSpanId != nil {
  apiSpan.SetParentSpanID(pcommon.SpanID([]byte(*apimTracePayload.ParentSpanId)))
 }

 // request start and end time are provided in unix milliseconds
 apiSpan.SetStartTimestamp(pcommon.NewTimestampFromTime(time.UnixMilli(apimTracePayload.RequestStartTime))) // example: OTel Start time     : 2024-12-10 14:26:32.435836303 +0000 UTC
 apiSpan.SetEndTimestamp(pcommon.NewTimestampFromTime(time.UnixMilli(apimTracePayload.RequestEndTime)))
 // ...
 attributes := apiSpan.Attributes()
 attributes.PutStr(conventions.AttributeHTTPTarget, apimTracePayload.HttpTarget)
 attributes.PutStr(conventions.AttributeHTTPMethod, apimTracePayload.HttpMethod)
 attributes.PutStr(conventions.AttributeHTTPHost, apimTracePayload.HttpHost)
 attributes.PutStr(conventions.AttributeHTTPClientIP, apimTracePayload.HttpClientIp)
 attributes.PutInt(conventions.AttributeHTTPStatusCode, int64(apimTracePayload.HttpStatusCode))

We extract the span and trace id of the traceparent header. The header format is defined in the format version-trace_id-parent_id-trace_flags (spec). An example is 00-a9b7d8bbcecf8c7d9d24c69cc86f694d-2defb8587ed6dfef-01.

Additionally, we set the ParentSpanId if one is present in the event data. This way the tracing backend can display dependencies and hierarchy between the different spans.

Starting the Receiver

The next step is to implement the listening logic for the receiver.

Since we want to connect to Azure Event Hubs, we can use the Go SDK (github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs) and follow the Azure instructions for setting up the connection:

func (apimtracerRcvr *apimtracerReceiver) Start(_ context.Context, host component.Host) error {
 apimtracerRcvr.host = host
 ctx := context.Background()
 ctx, apimtracerRcvr.cancel = context.WithCancel(ctx)

 tokenCredential, _ := azidentity.NewDefaultAzureCredential(&azidentity.DefaultAzureCredentialOptions{})
 checkClient, err := container.NewClient(apimtracerRcvr.config.StorageContainerUrl, tokenCredential, &container.ClientOptions{})
 if err != nil {
  return err
 }
 // create a checkpoint store that will be used by the event hub
 checkpointStore, err := checkpoints.NewBlobStore(checkClient, nil)
 if err != nil {
  return err
 }

 consumerClient, err := azeventhubs.NewConsumerClient(apimtracerRcvr.config.FullyQualifiedNamespace, apimtracerRcvr.config.EventHubName, apimtracerRcvr.config.ConsumerGroup, tokenCredential, &azeventhubs.ConsumerClientOptions{})
 if err != nil {
  return err
 }
 defer consumerClient.Close(ctx)

 processor, err := azeventhubs.NewProcessor(consumerClient, checkpointStore, &azeventhubs.ProcessorOptions{})
 if err != nil {
  return err
 }
 //  for each partition in the event hub, create a partition client with processEvents as the function to process events
 dispatchPartitionClients := func() {
  for {
   partitionClient := processor.NextPartitionClient(context.TODO())

   if partitionClient == nil {
    break
   }

   go func() {
    err := apimtracerRcvr.processEvents(partitionClient)
    if err != nil {
     apimtracerRcvr.logger.Error(fmt.Sprintf("Error processing events: %v", err))
     return
    }
   }()
  }
 }
 // run all partition clients
 go dispatchPartitionClients()

 processorCtx, processorCancel := context.WithCancel(context.TODO())
 defer processorCancel()

 if err := processor.Run(processorCtx); err != nil {
  return err
 }

 return nil
}

We use the DefaultAzureCredential struct. You can either login with the AzureCLI if running locally or provide credentials via environment variables.

The listener will create a connection to a Event Hub and waits for messages. Once an event is received the processEvents(partitionClient) function is called, which then maps the event to a trace:

func (apimtracerRcvr *apimtracerReceiver) processEvents(partitionClient *azeventhubs.ProcessorPartitionClient) error {
 defer closePartitionResources(partitionClient)
 for {
  receiveCtx, receiveCtxCancel := context.WithTimeout(context.TODO(), 1*time.Minute)
  events, err := partitionClient.ReceiveEvents(receiveCtx, 100, nil)
  receiveCtxCancel()

  if err != nil && !errors.Is(err, context.DeadlineExceeded) {
   return err
  }
  apimtracerRcvr.logger.Info(fmt.Sprintf("\nProcessing %d event(s)", len(events)))
  for _, event := range events {
   apimtracerRcvr.logger.Debug(fmt.Sprintf("Trace received with body %v\n", string(event.Body)))
   trace, err := mapToTrace(string(event.Body))
   if err != nil {
    apimtracerRcvr.logger.Error(fmt.Sprintf("Error mapping trace: %v . Ignoring the trace", err))
    continue
   }
   err = apimtracerRcvr.nextConsumer.ConsumeTraces(receiveCtx, *trace)
   if err != nil {
    return err
   }
  }
  if len(events) != 0 {
   if err := partitionClient.UpdateCheckpoint(context.TODO(), events[len(events)-1], nil); err != nil {
    return err
   }
  }
 }
}

Most of the code is wiring and logging. The important part is the call of the mapToTrace(string(event.Body)) call.

The mapped trace is then passed on to the next consumer in the OpenTelemetry collector execution chain with apimtracerRcvr.nextConsumer.ConsumeTraces(receiveCtx, *trace). After that the normal OpenTelemetry collector logic can be used, e.g. for batching, mapping and exporting to a OpenTelemetry backend.

Implementation Summary

The number of self-written lines of code is very low. In total it should be less than 200 lines of code. The rest of the code is generated using the awesome OpenTelemetry tooling.

We can now build our custom collector using a Dockerfile and have a executable version of our collector with the Azure API-Management tracing receiver.

The complete code with a Dockerfile for the example is available here: https://github.com/patst/azure-apim-otel-receiver/tree/main

Creating The API-Management Policy

Next we want to implement the API-Management policy which allows us to send traces to Azure Event Hubs.

Policies use razor templates and some predefined methods and objects can be used in the templates. For our implementation we need these policies:

• set-variable: To pass on variables between different steps in our policy
• set-header: To modify the HTTP Headers for a request, in order to make sure to pass on a traceparent header to the backend.
• authentication-managed-identity: Authorizes the API-Management to our Event Hub to send events (You must grant the API-Management system identity the required permission!).
• send-one-way-request: To asynchronously send events to Event Hubs using the REST-API using the previously fetched access token as authorization parameter.

The first step is to check, if there is already an existing traceparent header in the request.

1. header exists: We extract the traceId and reuse it. We only create a new spanId for our request.
2. header doesn’t exists: We create a complete new traceheader string

In both cases the result is saved in the traceparent header (existing header is overwritten, if it already existed).

Additionally we save the parentSpanId for later, to pass it on in the event data to the OpenTelemetry receiver. Code Snippet:

  <inbound>
    <set-variable name="parent_span_id" value="@{ string traceparent = (string)context.Request.Headers.GetValueOrDefault("traceparent", ""); if (traceparent == "") { return null; } else { return traceparent.Split('-')[2]; } }" />
    <set-header name="traceparent" exists-action="override">
      <value>@{
        const string allowedChars = "abcdefghijkmnopqrstuvwxyz0123456789";
        int stringLength=16;
        Random rand = new Random();
        char[] spanIdChars = new char[stringLength];
        for (int i = 0; i < stringLength; i++) {
        spanIdChars[i] = allowedChars[rand.Next(0, allowedChars.Length)];
        }
        // Check if the trace_id already exists
        string traceparent = (string)context.Request.Headers.GetValueOrDefault("traceparent", "");
        string traceId = "";
        if (string.IsNullOrEmpty(traceparent)) {
          traceId = Guid.NewGuid().ToString().Replace("-", "");
        } else {
          traceId = traceparent.Split('-')[1];
        }
        return "00-" + traceId + "-" + new string(spanIdChars, 0, stringLength) +"-01";
        }</value>
    </set-header>
    <base />
  </inbound>

The next part of the policy is active after the request has been executed ( outbound part of the policy).

We get an access token for the Event Hub and use the fire-and-forget policy to not block the normal API request execution longer than required.

In the set-body attribute we create the JSON payload for the event.

  <outbound>
    <authentication-managed-identity resource="https://[eventhub-namespace-name].servicebus.windows.net" output-token-variable-name="eventHubAccessToken" ignore-error="false" />
    <send-one-way-request mode="new" timeout="10">
      <set-url>https://[eventhub-namespace-name].servicebus.windows.net/[eventhub-name]/messages</set-url>
      <set-method>POST</set-method>
      <set-header name="Authorization" exists-action="override">
        <value>@((string)context.Variables["eventHubAccessToken"])</value>
      </set-header>
      <set-body>@{
        return new JObject(
        new JProperty("api_name",context.Api.Name),
        new JProperty("product_name",context.Product.Name),
        new JProperty("traceparent",context.Request.Headers.GetValueOrDefault("traceparent", "")),
        new JProperty("parent_span_id",(string)context.Variables["parent_span_id"]),
        new JProperty("http_target",context.Request.Url.Path),
        new JProperty("http_method",context.Request.Method),
        new JProperty("http_host",context.Request.Url.Host),
        new JProperty("http_status_code",context.Response.StatusCode),
        new JProperty("http_client_ip",context.Request.IpAddress),
        new JProperty("request_start_time",((DateTimeOffset)context.Timestamp).ToUnixTimeMilliseconds()),
        new JProperty("request_end_time", ((DateTimeOffset)System.DateTime.UtcNow).ToUnixTimeMilliseconds()),
        new JProperty("subscription_name",context.Subscription.Name)
        ).ToString();
        }</set-body>
    </send-one-way-request>
    <base />
  </outbound>

The payload contains exactly the attributes we have been implementing in our custom OpenTelemetry receiver.

The complete policy definition can be found in the GitHub repository (source).

Required Azure infrastructure

This article does not provide a detailled documentation how to create the required infrastructure components in Azure. A list of stuff you need to provision:

• Azure API-Management service (Azure docs)
• Azure Event Hub Namespace, Event Hub and consumer group (Azure docs)
• Permit the API-Management Service SystemAssignedIdentity to publish events in your event hub (assign Azure Event Hubs Data Sender role)

Summary And Next Steps

With the above described steps it is possible to send traces for requests received in the Azure API-Management to any OpenTelemetry compatible backend. It can be easily integrated in already existing OpenTelemetry collector architectures.

In general, it would be awesome to not be required to build this as a customer, but get it of-the-shelves from Azure.

The code still needs some cleanup, test cases and error handling to make sure it is really production ready. Nevertheless, it can be a starting point for further development.

Sources

• Example code repository with policy and receiver code: https://github.com/patst/azure-apim-otel-receiver
• Azure API-Management docs: https://learn.microsoft.com/en-us/azure/api-management/api-management-key-concepts
• OpenTelemetry custom receiver tutorial: https://opentelemetry.io/docs/collector/building/receiver/

Send traces from Azure API-Management to OpenTelemetry Collector was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Shift-Left in your deployment process: Use Kubectl together with Azure OpenAI to explain errors in Kubernetes deployment manifests before they are deployed!

Internal Developer Platforms (IDPs) have become invaluable tools for developers and organisations. They streamline and standardise application development and deployment.

Many of them are based on Kubernetes. Another pattern which is on the rise and which I would call the new standard for infrastructure and application deployment is the GitOps pattern.

With Flux and ArgoCD the CloudNativeComputingFoundation has two excellent graduated tools available for this.

The (simplified) flow looks like this:

This article will describe one of the challenges I observed. In addition, I describe how to use OpenAI to improve the developer experience by explaining the error messages that occur.

The Challenge

Having all the tooling implemented I took the time to look at the Git commit history of the infrastructure repositories. What I found, was a lot of commit messages like “fix deployment”, “fix 2”, “this time it works (really!)”.

Seeing this it became clear that one important puzzle piece is still missing in this GitOps process: Change validation!

Being a software developer I am very used to checks in Pull Requests, e.g. check if CI tests are still working, the application compiles, etc.

But in our current GitOps process we have been doing: Nothing!

How to get out of the “trial-and-error”-cycle?

The solution will depend on the tooling you are using, but the goal we have is to get feedback about the changes before they are applied by ArgoCD or Flux.

> We could call it shift-left for infrastructure changes.

Using Kubernetes, everything ends with a kubectl apply. Regardless, if you are using Helm Charts, plain Kubernetes YAML or Kustomize: You will be able to generate Kubernetes resource definitions in YAML.

We will use this common denominator together with the kubectl --dry-run feature to get the validations we need.

The process will look like this:

1. developer/operator makes changes in a feature branch of the infrastructure repository
2. CI pipeline will template the changes (=> generate Kubernetes resource YAML)
3. templated YAML will be sent to the Kubernetes API-Server in--dry-run mode
4. we will save any error message in a file

Implementation

Before we start, these are the prerequisites:

• Kubectl installed
• (read-only) user with access to the Kubernetes clusters
• Helm installed

> In this step I assume you are using Helm Charts. If you are using something else, just make sure that you will have file(s) with YAML which can be send to the Kubernetes API-Server.

Make sure you are in the directory containing the helm chart and run:

# set the Kubernetes context to the namespace, the changes should be applied to later
helm template "my-release" . --debug  > templated.yaml

Did you saw it? You already have your first check implemented!

“helm template” will fail if the syntax of your helm chart is not correct. In that case helm will return an exit code of “1” and your pipeline will fail. Our first success is, developers can no longer accidentally merge syntactical wrong definitions!

> You can save the helm exit code in a variable with HELM_EXIT_CODE=$? to implement better error handling

Next step is to send the YAML to the Kubernetes API-Server to get further feedback:

# set the Kubernetes context to the namespace, the changes should be applied to later 
kubectl apply --dry-run=server -f app.yaml 2> validation_errors.txt || echo "Validation failed"

It is important to use --dry-run=server. The Kubectl docs describe the difference:

dry-run: Must be “none”, “server”, or “client”. If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.

Running in server mode makes sure the API-Server validates the request, because webhooks inside the cluster are called. Some of the errors I observed frequently:

• using a wrong StorageClass
• setting a wrong SecurityContext (e.g. runAsNonRoot: false)
• …

Make sure to append the content of the file validation_errors.txt written above to the Pull Request as comment. This improves developer experience, because the error is directly visible inside the Pull Request and users do not have to open the pipeline runs output.

Improve error messages with Azure OpenAI

Instead of just presenting developers with an error message, it’s better to provide them with a solution. Sometimes, Kubernetes error messages can be confusing and the solution might not be immediately clear.

Thanks to advancements in AI, we can use an LLM to do the heavy lifting. In this example, we use Azure OpenAI, but it will work similarly with vanilla OpenAI or any other LLM.

Before we start, these are the prerequisites:

• Azure OpenAI deployment (e.g. with GPT-3.5 or GPT-4)
• Azure OpenAI API Key
• cURL
• jQ

We will use the output generated in the step above (validation_errors.txt) and add another step to our validation pipeline.

We will create a payload for the Azure OpenAI service, which includes:

• a system message setting the context for what we are trying to do
• the error messages of our helm dry-run

request=$(jq --arg errorMsg "$(cat validation_errors.txt)" -n '{
  "messages": [
    {"role": "system", "content": "Your are a helpful assistant. \n Context: \n- You are answering question about kubernetes deployments\n - Each answer ends with a joke about Kubernetes \n - You get error messages from the Kubernetes API Server \n - you provide YAML code snippets how to fix the error"},
    {"role": "user", "content": $errorMsg}
  ],
  "max_tokens": 800,
  "temperature": 0.9,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "top_p": 0.95,
  "stop": null
}')

The complete API description can be found in the Azure OpenAI REST API reference.

The next step is to create a REST call and send the payload to Azure OpenAI. Then we extract the text from the response message using jQ:

# make sure to replace "<your-openai-instance-name>" with the name of your instance
deploymentName="gpt-4" # make sure the deployment exists in your instance
apiKey="" # set your api-key
curl "https://<your-openai-instance-name>.openai.azure.com/openai/deployments/${deploymentName}/chat/completions?api-version=2023-12-01-preview" \
    -H "Content-Type: application/json" \
    -H "api-key: $apiKey" \
    --output response.json \
    --data "$request"
jq -r '.choices[0].message.content' response.json > output.txt
# add the content of "output.txt" to your pull request!

Make sure to add the content of output.txt to your pull requests as a comment.

This process will help developers understand the error messages better and find solutions more quickly, enhancing productivity and efficiency.

To improve the feedback, you can play around with the system message template.

Feel free to share your experiences in the comments!

Further reading:

• Azure OpenAI docs — https://learn.microsoft.com/en-us/azure/ai-services/openai/
• Helm template docs — https://helm.sh/docs/helm/helm_template/
• Prompt engineering techniques — https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions
• ArgoCD: https://argo-cd.readthedocs.io/en/stable/

GitOps and AI: Explaining Errors in CI before applying them was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Helpful instructions to help you on your Observability journey

OpenTelemetry has established itself as the standard for Observability after emerging out of the OpenTracing and OpenCensus standards. It tries to standardise different kinds of Signals (logs, traces, metrics). The Observability signal relevant to this article is traces.

To better understand, it’s worth reading the basic OpenTelemetry concepts.

We use Grafana Tempo as the backend for our traces. It allows us to store the trace information in an inexpensive object storage instead of hosting an ElasticSearch database when using a tool like Jaeger.

Because we are providing shared Kubernetes clusters for application development teams, another very important feature of Grafana Tempo is multi-tenancy.

Basic Flow

Each development team has a set of Kubernetes namespaces with a tenant_id label. The applications are boosted with an OpenTelemetry library (see the list of supported languages at this link).

Traces originate in the applications, then are sent to an OpenTelemetry collector installed in the cluster with the awesome OpenTelemetry Kubernetes Operator. A collector pipeline has three main parts:

• Receivers are components that accept traces sent from workloads. We use the built-in otlp receiver, which accepts traces send via gRPC. A collector can accept traces from different sources using different receiver configurations.
• Processors can transform, filter, or route traces. We use the processor to enrich traces with information about Kubernetes Pod and Namespace names of the sending workloads and extract a tenant id from a Namespace label.
• Exporters send traces to backends like Grafana Tempo. Here we use the otlp exporter via gRPC.

The collector pipeline for the workloads extracts the tenant id from a Namespace label using the k8sattributes processor. (The processor requires some permissions to query the Kubernetes API server. See the processor's README file.)

receivers:
  otlp:
    protocols:
      grpc:
processors:
  k8sattributes/default:
    extract:
      labels:
      - tag_name: tenantId
        key: "tenant-id"
        from: namespace
  routing:
    from_attribute: tenantId
    attribute_source: resource
    table:
    - value: tenant1
      exporters: [logging, otlp/tenant1]
    - value: tenant2
      exporters: [logging, otlp/tenant2]
exporters:
  otlp/tenant1:
    endpoint: tempo.grafana.svc.cluster.local:4317
    tls:
      insecure: true
    headers:
      X-Scope-OrgID: tenant1
  otlp/tenant2:
    endpoint: tempo.grafana.svc.cluster.local:4317
    headers:
      X-Scope-OrgID: tenant2
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes/default, routing]
      exporters: [ otlp/tenant1, otlp/tenant2]

Next, we define an exporter for each tenant we have. They all configure the same endpoint URL for Grafana tempo but have a different X-Scope-OrgID header value. The routing processor routes traces to the different exporters depending on the tenant id.

This is a bit cumbersome, but there is no way to use variables inside the headers block for the otlp exporter. This would enable us to have only one exporter entry with variable content in the X-Scope-OrgID header.

Ingress-Nginx and OpenTelemetry

Ingress-Nginx has supported OpenTelemetry since version 1.7.0 (released on March 24th, 2023. See the pull request #9062). We were looking forward to this new feature because it allows the collection trace information not only at the workload level but already one step earlier when the traffic enters the Kubernetes cluster.

To enable the OpenTelemetry integration using the Ingress-Nginx helm chart, just add these parameters:

controller:
  config:
    enable-opentelemetry: "true"
    opentelemetry-config: "/etc/nginx/opentelemetry.toml"
    opentelemetry-operation-name: "HTTP $request_method $service_name $uri"
    opentelemetry-trust-incoming-span: "true"
    otlp-collector-host: "otel-collector.grafana.svc.cluster.local"
    otlp-collector-port: "4317"
    otel-max-queuesize: "2048"
    otel-schedule-delay-millis: "5000"
    otel-max-export-batch-size: "512"
    otel-service-name: "nginx-proxy" # Opentelemetry resource name
    otel-sampler: "AlwaysOn" # Also: AlwaysOff, TraceIdRatioBased
    otel-sampler-ratio: "1.0"
    otel-sampler-parent-based: "false"

While the workloads are always running inside a Kubernetes namespace, which has the tenant id attached as a label, the Ingress-Nginx controller is a shared deployment operated by the platform team, and the tenant information cannot be extracted from the namespace.

We had to extract the tenant information from another location to route the traces originating in the ingress controller to the correct Grafana Tempo tenant.

Ingress-Nginx OpenTelemetry directives

While the Ingress-Nginx controller documentation currently falls short, we found the required information in the module which is used for the OpenTelemetry part. It is located in the following GitHub project.

We were able to find the directives which are provided by the OpenTelemetry module. Especially the directive opentelemetry_attribute was very interesting for us. It allows us to add custom attributes to spans.

We configured the ingress controller to add these directives to each server block in the nginx config. The configuration example below is for the helm chart:

controller:
  config:
    server-snippet: |
      opentelemetry_attribute "ingress.namespace" "$namespace";
      opentelemetry_attribute "ingress.service_name" "$service_name";
      opentelemetry_attribute "ingress.name" "$ingress_name";
      opentelemetry_attribute "ingress.upstream" "$proxy_upstream_name";

After making this configuration change, traces sent from the ingress controller contain information about the Ingresses namespace, name, and where it routes traffic to.

But where to get the tenant id from? Lucky for us, all of our namespaces follow a naming schema like tenantId-<namespaceName>. We can extract the tenant id from the namespace attribute attached to the spans.

The collector pipeline above is changed to make this possible. Here’s what the code looks like:

receivers:
  otlp:
    protocols:
      grpc:
processors:
  k8sattributes/default:
    extract:
      labels:
      - tag_name: tenantId
        key: "tenant-id"
        from: namespace
  # in case of ingress-nginx traces the ingress namespace is in the attributes.
  # we need to extract the account id from there and eventually transfer it to the resources
  # see Resource: https://opentelemetry.io/docs/concepts/glossary/#resource
  # vs Attribute: https://opentelemetry.io/docs/concepts/glossary/#attribute
  attributes/ingress:
    actions:
    - key: "ingress.namespace"
      pattern: "(?P<temp_tenant_id>.{6})"<<< 
      action: extract # this overwrites existing keys
  groupbyattrs: # copy the attribute containing the tenant id to the resource attribute which is used for routing
    keys:
      - temp_tenant_id
  resource/ingress:
    attributes:
    - key: "tenantId"
      from_attribute: temp_tenant_id
      action: insert
    - key: "temp_tenant_id"
      action: delete
routing:
  from_attribute: tenantId
  attribute_source: resource
  table:
  - value: tenant1
    exporters: [logging, otlp/tenant1]
  - value: tenant2
    exporters: [logging, otlp/tenant2]
exporters:
  otlp/tenant1:
    endpoint: tempo.grafana.svc.cluster.local:4317
    tls:
      insecure: true
    headers:
      X-Scope-OrgID: tenant1
  otlp/tenant2:
    endpoint: tempo.grafana.svc.cluster.local:4317
    headers:
      X-Scope-OrgID: tenant2
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes/default, attributes/ingress, groupbyattrs, resource/ingress, routing]
      exporters: [ otlp/tenant1, otlp/tenant2]

Using this pipeline, the namespace information of the Ingress-Nginx is used. The tenant id is extracted with a regular expression and stored in a temporary attribute. Then the attribute is copied to the resources using the groupbyattrs processor.

This is required because OpenTelemetry has the concept of attributes and resources. The k8sattributes processor stores the information in the resources part. We use these attributes already for routing (this is why we configured attribute_source: resource).

This is a bit ugly, but we needed help finding a better way to do this. If you have ideas feel free to put suggestions in the comments!

Next Steps

The collector pipeline above can be further optimised by using the following:

• the batch processor for sending traces batched
• the tail-sampling processor to filter out traces for endpoints like /actuator or /healthwhich are called very frequently by the Kubernetes API server but bring no real value to be recorded in Grafana Tempo

Grafana Tempo and the OpenTelemetry collector usage are very new for us now. If any important points are missing above and you see further optimisations leave a hint in the comments.

Hopefully, the instructions help you as well on your Observability journey!

Further Read

• OpenTelemetry.io
• OpenTelemetry: Up and Running
• Grafana Tempo
• Ingress-Nginx OpenTelemetry

OpenTelemetry: Sending Traces from Ingress-Nginx to multi-tenant Grafana Tempo was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

We cannot use the Microsoft hosted agents, because all of our resources are running in private virtual networks and are not accessible from the hosted agent network. This means we use self-hosted agents, which are running in a Kubernetes cluster.

They are automatically scaled up and down depending on how many build jobs are currently queued. For autoscaling we use the KEDA azure-pipelines scaler.

Using the build agents got more and more popular among the development teams the last months. This led to the appearance of some problems and the discovery of new requirements. We needed to make adjustments, especially as a reaction to teams with more monolithic architectures.

Initial Setup

The build agents are running as Deployment inside a Kubernetes cluster and are connected to a ScaledObject provided by KEDA.

The build agent replica count is increased and decreased by KEDA, depending on the amount of build jobs queued. This only works if enough resources are available inside the Kubernetes cluster. (=> This means a Kubernetes Node is available on which a new build agent can be started.)

We get the best resource utilisation when the cluster reacts to changes of resource requests. For that we use the cluster autoscaler.

The cluster autoscaler observes the resource requests and scales up new Kubernetes Nodes, if more resources are requested than available. On the other side it scales down Nodes if there are too many unused resources in the cluster.

In the beginning, we scheduled the build agents with resource requirements as described below:

...
resources:
  requests:
    cpu: "1"
    memory: 1024Mi
  limits:
    # cpu:  no CPU limit!
    memory: 4096Mi
...

This setup worked fine for a long time, because we had many short running jobs, which only consumed resources for a short time. After onboarding larger applications on the platform we observed the first problems.

Proceeding in the description of these problems I assume you are familiar with Kubernetes Resource Management for Pods and Containers.

Out-of-memory killed build agents

Our alerting showed Kubernetes Pods getting OOM killed (Exit Code 137). The reason was easy to detect: Compiling a large Java project and executing many JUnit tests just consumed more than 4GB of RAM, which was defined as memory limit. The Kubernetes scheduler reacts with restarting the Pod. Each time this happens, the build job will show as failed in Azure DevOps and must be restarted. This is a poor Developer Experience.

The initial reflex was to increase the build agents memory limit to 8 GB RAM. But while fixing the issue above this surprisingly led to a set of new problems.

Too low memory requests

Some build pipelines are spawning parallel build jobs with Java compiling, Unit- and Integration-Testing and more. Since we had a very low memory request of 1 GB the Kubernetes scheduler placed multiple build agents running these jobs on one Node.

Let’s assume the Kubernetes Node has a VM Type with usable resources of 6 vCPUs and 16 GB RAM. In theory more than four build agents can be placed on the Node, if we take only the requests into consideration. And that is exactly what happened.

The outcome where two bad situations:

1. The builds jobs saturated the Nodes complete CPU resources while compiling Java code. Things got very slow and build jobs timed out because of CPU starvation.
2. If the build jobs used a lot of memory (remember the new memory limit of 8 GB) they used the whole RAM the Node has. This resulted in swapping and the Kubernetes scheduler started evicting Pods due to a Memory pressure event. (See the docs for Node out of memory behaviour)

We got a lot of alerts from our monitoring and thought about possible solutions. The obvious fix was increasing the CPU and memory requests for the build agents. Now the Kubernetes scheduler placed less agents on one Node. Additionally, we configured (soft)anti-affinitieson the build agents Pod to get them scheduled on different Nodes if possible (See K8S Docs Assigning Pods to Nodes).

Bad resource utilisation

What we disliked about this solution was a decreasing resource utilisation. Now less build agent Pods where scheduled on a Node. But only some of our build jobs are really requiring these extra resources added by increasing the requests and limits. (A solution will be described in the “different agent types” section later)

Having less jobs run on a Node made the cluster autoscaler becoming more active scaling up and down Nodes depending on queued jobs. This led to another two interesting error scenarios. (*sigh*)

Killed build jobs

We again started seeing messages like “We stopped hearing from the build agent” in the Azure DevOps build logs. The builds seemed to just stop with no special error message in the logs.

This gave us some headaches. First we expected some network problems, but found no proof for that.

The cancelled build jobs did not consume excessive resources and the Node showed no special events as well. The build jobs with errors where randomly distributed across all customers. It seemed that longer running jobs are more often killed. This was a very interesting finding.

Our build agents pick up jobs which can run for a long time (e.g. over 40 minutes). When KEDA decides to scale down a build agent the Pod receives a SIGTERM event and is put into Terminating state.

The Kubernetes scheduler waits for the time specified in terminationGracePeriodSeconds (default: 30 seconds) and if the Pod is not stopped after this time it receives a SIGKILL event and is stopped immediately. If a build job is still running it won't finish and we see an error message like "we have stopped hearing from the build agent".

Our solution was to increase the terminationGracePeriodSeconds to 50 minutes. This is the longest time we guarantee the application teams a build job can run, even if in 99% of the cases longer builds will work.

Cluster autoscaler and max-graceful-termination-sec

Even after setting the terminationGracePeriodSeconds for Pods to a higher value we still experienced problems with killed build agents. This is related to the Kubernetes cluster autoscaler.

The cluster autoscaler checks the resource consumption of workloads and if Nodes have a low resource usage they are stopped and removed.

The default scale down threshold is 50% resource usage. That means if the Node has memory and CPU requests lower than 50% of its capacity it is a scale down target. Scale down will not happen immediately but after a scale-down-unneeded interval (default: 10 minutes). If the resource usage does not increase during that period the Node will be scaled down. Scale down happens in multiple steps:

• add a taint NoSchedule to the Node, no new workloads are placed on the Node
• send a SIGTERM event to all Pods to terminate them (they are rescheduled on other Nodes if required)
• wait until all Pods are evicted or max-graceful-termination-sec is reached and they are killed
• delete the Nodes virtual machine

What does that mean for our build jobs?

When there is only one build agent left running on a Node, which usually has 6 CPU cores, the Node will get evicted. Since we have configured terminationGracePeriodSeconds to 50 minutes the build job may continue to run up to this time.

BUT the Node does not wait that long because max-graceful-termination-sec is only set to 5 minutes (default). This results in the Pod getting killed with a SIGKILL event anyway after that time.

Our solution was to increase the max-graceful-termination-sec for the cluster autoscaler to match the build agents terminationGracePeriodSeconds (= 50 minutes). This should be the worst case and most of the build jobs will finish earlier.

Things started looking good after all these changes.

Different build agent types in Azure DevOps

The last thing that needed improvement is the resource usage of the build agents. Until that point every build job had the same resource requests and limits. We as a platform team cannot decide which build job could cope with less resources and which job requires more. We needed a way to give the developers an option to chose the right build agent sizing for their jobs.

The plan was to have different build agent Deployments with different limits (e.g. 4 GB memory and 8 GB memory). Next the developers had to choose which agent should execute their build job.

Azure DevOps has two different mechanism to select an agent for a build job:

1. Build Agent pools: Can be specified in the pipeline YAML. Agent pools must be added to an Azure DevOps project prior usage and pipelines need permission to run on a certain agent pool. (Permission can be granted for all pipelines in a project at once)
2. Demands + Capabilities: A self-hosted build agent has capabilities. These are either auto-detected for some tools if a certain binary (e.g. a Javac executable) is found in PATH or custom capabilities can be added by setting environment variables (we use this later). A pipeline can then specify demands to formulate requirements a build agent must have to be able to run a job.

Demands and capabilities were the feature we wanted to use because we are providing the build agents for the whole Azure DevOps Organisation and do not want to create a new agent pool and grant permission to all projects each time we add an agent type.

KEDA supports scaling with demands since Version 2.8 (introduced with #2795). We specified different triggers for the different agent types we have:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scaledobj-agent4gb
spec:
  scaleTargetRef:
    name: azagent-agent4gb
  pollingInterval: 30
  maxReplicaCount: 10
  minReplicaCount: 0
  triggers:
    - type: azure-pipelines
      metadata:
        targetPipelinesQueueLength: "0"
        poolName: "SELF-HOSTED-AGENTS"
        organizationURLFromEnv: "https://<ADO-ORG-URL>"
        demands: "limit4GB" 
      authenticationRef:
        name: pipeline-trigger-auth
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scaledobj-agent8gb
spec:
  scaleTargetRef:
    name: azagent-agent8gb
  pollingInterval: 30
  maxReplicaCount: 10
  minReplicaCount: 0
  triggers:
    - type: azure-pipelines
      metadata:
        targetPipelinesQueueLength: "0"
        poolName: "SELF-HOSTED-AGENTS"
        organizationURLFromEnv: "https://<ADO-ORG-URL>"
        demands: "limit8GB" 
      authenticationRef:
        name: pipeline-trigger-auth

The ScaledObjects all look the same apart from the demands part. They are all in the same Azure DevOps Agent Pool.

Important is that the Deployment to which scaleTargetRef points really has the capability specified as demand in the trigger. Otherwise the scaled up build agents will not be able to execute the jobs. (See the docs)

During testing this configuration looked good. We have a Helm chart for the Azure DevOps build agents and by adding a new entry to a list we can add/remove/reconfigure the available build agent types.

A customer then can select a certain agent type by adding this to a pipeline:

pool:
  name: 'SELF-HOSTED-AGENTS'
  demands: "limit8GB"

We are still thinking about the capability naming in order to be able to change things later. Maybe T-Shirt sizes would work better than specifying an exact number like ‘limit8GB’ for memory limits.

Too many agents scaled

After rolling out these changes we noticed an overall increase in build agent Pods running.

Since specifying a demand is optional and especially in the beginning most of our customers did not configure any demands a shortcoming in the current KEDA azure-pipelines scaler became clear:

It was possible to state, that a job only runs on agent with certain capabilities, BUT the trigger definition did not allow to configure that only jobs with certain demands should scale the definition.

Sounds complicated, but just think of the default case of no demands specified. This default will match all of our trigger definitions. If we have five different build agent triggers defined, each build job without demand is spawning five new Pods to run the build job. Of course only one will execute the build jobs and the others will be scaled down eventually, but this behaviour will lead to a lot of Nodes being scaled up and down.

We added a new configuration attribute called requireAllDemands to solve this issue. This was implemented in #4203 and will be rolled out in the next KEDA release (2.10).

With this configuration flag we are able to configure one trigger as default one (requireAllDemands: false) and all others will only be scaled if their demand is really requested by a build agent (requireAllDemands: true).

Outlook

Providing build agents sounded very simple in the beginning, but what this article hopefully made clear is that there are quite a few things to consider.

Having been running GitLab and Jenkins before, sometimes Azure DevOps felt a bit cumbersome to use and lacking features. But at the moment we are quite happy with our setup. We are running several thousand jobs per day on our build agents and rarely see any severe errors.

One area which requires our attention next is providing teams the opportunity to run containers in their build jobs. To do this they need a “docker-like-thing” without handing out root access or mounting a shared daemon. How we are solving this is probably worth an article in the future.

What are your experiences with Azure DevOps and self-hosted build agents? Any particular problems we still have not covered yet? Feel free to add them in the comments.

👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇

🚀Join FAUN Developer Community & Get Similar Stories in your Inbox Each Week

Pitfalls autoscaling self-hosted Azure DevOps build agents in Kubernetes was originally published in FAUN.dev() 🐾 on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the past years we moved from traditional operations to continuous deployment to have better automation and reduce the risk of errors. During that change applications where no longer deployed manually by people in the operation department, but automated by pipelines run by a CI server like GitLab, Jenkins or whatever the teams where using at the moment.

Traditional attack surfaces

In the traditional push based deployment world one of the main attack surface has been losing credentials for administrative accounts. This could be some server admin password for a virtual machine or application server. Getting those credentials would allow a potential attacker to change the machine configuration and deploy stuff which should better be not running in your environment.

In the traditional operation department based deployment model this risk was often enhanced by missing transparency about ‘who is using what password?’, which leads to situation where you pretty much never can rotate a password without the fear of breaking something. This was somehow solved by introducing password managers or pulling credentials from a central secret store.

Using continuous deployment with a tool like GitLab, Jenkins, Azure DevOps pipelines or Github actions was the automation of the manual repetitive deployment tasks. Using this it was possible to increase the amount of deployments by a huge factor and decreasing the time a feature needs to production drastically.

By using it suddenly the build server plays a very central role in your deployment process and needs to have your deployment credentials. Until then it was a tool to produce some kind of application binary and store it in a place where the operator can fetch and deploy it. Getting rid of this manual work (which is always a good idea!) suddenly a tool historically only used by developers needs be secured much better. In combination with build servers like Jenkins which tend to have a lot of plugins installed, it brings a lot of new challenges.

From imperative to declarative deployments

What both techniques had in common was the push principle: Whenever something new should be deployed somebody (operator or CI server) was running some commands to deploy the application to a virtual machine, application server or some other deployment target.

This process was imperative: The deployment manual or pipeline contains a sequential list of commands to be run for a deployment. In cases of errors it might be hard to continue at a certain point because partly successful deployments can be hard to be rolled back or existing resources cause problems.

Then the raise of declarative infrastructure came: Tools like terraform or cloud specific tools like CloudFormation (AWS) or ARM (Azure) templates allowed teams to describe their infrastructure and have a external software making sure this description gets somehow deployed and reconciled in subsequent runs.

Now not only new application versions could be deployed using automation but your whole infrastructure landscape could be deployed by running a pipeline.

The huge advantage over the traditional imperative deployment model was idempotency: It is very hard to make sure your pipeline works in subsequent runs using scripting in bash. Tools like terraform on the other hand have this functions already build-in. They can rerun deployments and if the declaration of your infrastructure did not change, then following deployments will recognise there is nothing to do.

From push to pull

After moving to a declarative description of our environments (application version + infrastructure) that next evolution is moving from deployment pipelines which do the deployments (push) to operators which are running inside an environment and watch your Git repositories for changes (pull).

This eliminates the risk of having deployment credentials in your pipelines. The operator is already running in the target environment and can therefore be much better secured.

Using Kubernetes as operation platform we see this pattern when we use operators for custom resources or by using Kubernetes at all: We create a Deployment object in Kubernetes and somebody is observing this and creating Pods, ReplicaSets, Volumes and other resources on our behalf.

GitOps tools like ArgoCD or FluxCD brought this pattern to a new level. We can watch Git repositories for changes to our helm charts, Kubernetes YAMLs or whatever we have and the changes are just getting installed in as many Kubernetes clusters as we like without any manual intervention.

This article is not about GitOps in general. Just read the awesome documentation at https://www.gitops.tech/ to learn more about that.

New attack surfaces

With great power comes great responsibility. (Spiderman? ;-) )

As we moved from manual deployments done by somebody, to a world where creation and destruction of our complete infrastructure is only one Git commit away, we have to make sure no bad actor is able to compromise our repositories.

Automating deployments moved the risk of failures from the people in the operation department to pipelines running on a build server. Now GitOps is shifting some risks one step further left to the Git repository. That is traditionally the place where developers are working on.

One of the main advantages of GitOps is the Git history where you can find a complete auditable trail of changes done to your repository (= your infrastructure). Enabling signing of commits using GPG makes sure you know who did the change at what time.

So far so good, but there is a catch: In case somebody gets too many permissions to your repository he can alter the history using git force push. As Git is a decentral version control system you can restore the history from a different workstation or server. But spotting the changed may not be that easy.

Final thoughts

While using the GitOps principles on a daily basis it was a nice look back where we came from and what did change in only a few years. Switching from the traditional or build server based deployments to GitOps shouldn’t be anything you are afraid of. (That was NOT the goal of this article ;-) )

Nevertheless, it is always a good approach to not only see the shiny new world with all its possibilities, but take a moment to reflect what might go wrong. Working with User stories this might be the right moment to think about an Evil User story.

In the last chapter we learned that the Git repository is something which should be secured well. The next article will focus how to secure a repository in Azure DevOps repositories.

The list of security issues was of course not exhaustive. If you see any important points missing feel free to add them in the comments.

In the last article we discussed the security challenges moving from a traditional imperative deployment model to a declarative GitOps model. For more details check out the article at https://medium.com/@pat-st/difference-in-security-challenges-while-moving-from-traditional-deployments-to-gitops-a2b7deac64a4 .

In the last article we discovered that our Git repository is a new threat to our security. Now we will learn how to configure Azure DevOps repositories to be secure.

Limit access to repository

Of course many problems we describe below could be solved by just denying access to our repositories to most people. This seems like the easy solution, but personally I like transparency in the organisation because that allows developers to learn from each other and fosters collaboration between different teams.

If Team A has a certain problem Team B already solved it can find the solution just by looking at their code. If a team sees a error in the repository of another team it even can just create a pull request to make the code better.

What I propose as ideal solution is to give everybody in the organisation at least read access to your repositories and deny unauthenticated access. This will not expose your repositories publicly but enables teams learn what is already available in their Azure DevOps organisation.

Preventing unapproved changes

One risk factor we discovered are unapproved changes to our Git history, e.g. git force pushes which rewrite the history. To reduce the risk of that happening we will protect the main branch in our git repository. Details about that can be found in the Azure DevOps docs at Git branch policies.

Above screenshot shows the configuration for the main branch. Important is to NOT check the checkbox ‘Allow requestors to approve their own changes‘. Otherwise we would forfeit the purpose of this configuration, because one person alone would be able to merge changes.

With this minimal change we achieved:

• no direct changes to main branch allowed
• apart from the pull request creator at least a second person is required to complete the pull request
• when new changes are pushed a new approval is required

There are more settings to fine tune the behaviour. You can take a special look what should happen if new changes are pushed. You must find the right balance between security requirements and developer experience. If it takes a long time to get the required approvals to get a pull request merged this will be bad for your turnaround times and will demotivate the developers.

Having pull requests for every change opens up a whole new set of possibilities:

• you can create additional automated checks like running tfsec or terrascan for terraform files or some CVE checker for your Java application. The check results can directly be pushed as comments to your pull request and help reviewers to get a clear picture about the changes in a pull request.
• you can have pull requests created by tools like Renovatebot to get dependency versions updated automatically. We use this feature very extensively to update all kind of dependencies (terraform modules, CLI Tool versions, …)
• discuss about documentation changes or architecture decision records with other stakeholders directly in the pull request
• knowledge transfer because at least a second pair of eyes has seen each code change

In my experience having pull requests is a very good tool to work together and make informed decisions. It is not always about solving every issue in the pull request, but about creating new tasks or user stories for fixing some technical debt you may introduce by purpose.

Very important is to keep your branches shortlived! Nobody wants to review changes to hundreds or thousands lines of code on the one hand and on the other hand no developer should work on branches which live for several days. Otherwise merge conflicts will arise and productivity will decrease!

Deployment repository structure

Now taking a look at a infrastructure repository the layout may look like this:

We have different folders for different environments. There are different strategies to describe environments, e.g. using Git branches or dedicated repositories for different environments. In the last years I tried out each and every of these strategies and (in my opinion) the only working option is using directories as representation of environments.

Now we want to give the developers more freedom to make changes to DEV and UAT environments and have stricter security rules for the PROD environment.

The problem: Git does not know about permissions in general and for folders in particular.

The solution to this challenge is different for every platform, but usually you can configure different conditions for different path changes. In Azure DevOps we can solve this using reviewer policies:

We can define different reviewer policies for our different stages. This way we can enforce approval from different groups per stage. We use this feature to enforce approval for production environments from application owners while DEV and UAT approvals can be given be the developers themselves.

The good thing in Azure DevOps: The approval groups can be backed up by Azure Active Directory groups. That way we can use the standardised company legitimation processes to gain access for deployments to different environments.

Build validation and status checks

Now we have secured who is allowed to make changes to our repositories. The next step is to increase the confidence of merging changes. Sometimes just seeing some deployment YAML files or Terraform HCL files does not really helps to find out what this merged pull request might to in the end.

The solution is to make dry-runs for your changes. How that exactly looks like is depending on your deployment tools:

• Terraform: Just run a terraform plan for the changes in your pull request. Terraform will come up with a diff between your current state and the state described in your branch. (Hint: Don’t bother to use the Microsoft Terraform extension for Azure DevOps. It is not maintained. Instead use the excellent Extension by Charles Zip called Azure Pipelines terraform tasks). You can have a pipeline to create the terraform plan and allow the merge of the pull request only if the pipeline succeeds. In addition to that the created terraform plan can be viewed to spot accidental changes. We created a custom pipeline step which creates a short summary of the plan and pushes it as comment to our pull request.
• Helm Charts: A good start is to display the diff between the values.yaml files you may have. As you will learn at some point this is only one part of the story, especially while updating helm chart versions of third party charts. In addition to that you need to run a helm template command on the main branch and your feature branch to create a meaningful diff. This allows you to preview the changes which will really be happening on a Kubernetes cluster!

In addition to that you should run security testing tools like described above, e.g. tfsec or terrascan for Terraform.

Know your committer

Git has the nice feature to allow you to chose any email address in your commit message. So in the end you cannot be really sure that <Mike.Anybody@somemail.yxz> is really who he claims to be. This is really bad if you want to keep an auditable trail of changes to your repositories.

Fortunately there is a (more or less) easy solution to that: Signing of commits. Each committer has a SSH or GPG keys and can sign his commits with this key. Knowing the public signatures of these keys allows to verify that they are coming from known and trusted sources.

Unfortunately at this point Azure DevOps is not able to enforce the commit signing. There is very old User Voice Request for this with disappointing feedback until now.

Final thoughts

Azure DevOps does offer a lot more configuration properties to control who is allowed to do what on a repository. In my experience the above described configurations will bring you a lot of security without sacrificing to much usability. The status checks and build validations even add a lot of confidence while doing changes to your applications.

Being an application developer a lot of this does not look very new and we applied great parts of it already in our development processes in the last decade. That is what is most interesting about it:

Operations is just another style of development nowadays.

While maturing as a team you will find new status to add to your pipeline or even retire some if they bring no use. What is important to always give transparence what is happening at the moment and what is required to get something merged. Make sure your pipelines do not get too big, this might lead to unmanageable script monsters. ;-)

If you are missing something very important feel free to add a hint in the comments.

Secure your GitOps repository in Azure DevOps was originally published in System Weakness on Medium, where people are continuing the conversation by highlighting and responding to this story.

Create KeyVault secrets without storing them in the Terraform state to allow an automated external rotation later.

We are heavily users of Terraform and most of our infrastructure is deployed in Azure.

Naturally we have places where we need to create and store secrets in an Azure KeyVault. Unfortunately the Azure KeyVault offers no option to create and auto-rotate secrets.

In addition to just creating a secret we want a rotation process where we can change a password (e.g. for a Postgres database) without any downtime in the consuming application.

For Postgres this can be done using different roles which inherit their permissions from a group role. There are some good descriptions like this one with more details.

Baseline: We need multiple secrets which are rotated at different times. The secrets values must be usable in Terraform.

# Naive solution

What we did first was using the Terraform random provider to generate a password and then use the azurerm_keyvault_secret resource to store the password in a KeyVault:

data "azurerm_key_vault" "example" {
  name                = "mykeyvault"
  resource_group_name = "some-resource-group"
}

resource "random_password" "admin-pw" {
  length           = 32
  special          = true
}

resource "azurerm_key_vault_secret" "admin-pw" {
  name         = "admin-pw"
  key_vault_id = azurerm_key_vault.example.id
  value        = random_password.admin-pw.result
}

This works quite well for initial creation of secrets. But there are some downsides to this approach:

First the generated password is stored in plain text in your Terraform state (you better don’t lose you state files anyway!).

Specifying an expiration can be difficult because you need a datasource with a date and time which does not change on every terraform plan execution. Otherwise you will see constant changes in you Terraform pipeline. You can try to achieve that to a certain extent using the time_rotating resource provided by Hashicorps Time Provider.

Implementing something like having two passwords where one is active and the second can be rotated if it is not used anymore will give you a real headache.

This ‘rolling password update‘ is not something you want to solve in Terraform. You will end up with a lot of dependent time resources, conditional blocks where you are trying to sort out which change should trigger which resource updates and so on.

We managed to come quite far trying but as we have several Terraform modules with dependencies to each other we often ended up having some datasource or condition which can only be evaluated deferred during the apply phase. This leads to a lot of changes in Terraform plans.

# External password rotation

We ended up using a password rotation mechanism external to Terraform. Azure even offers some help in their ‘Automate the rotation of a secret for resources that have two sets of authentication credentials’ article.

This can only work if the password itself is not managed with a Terraform resource like the azurerm_keyvault_secret. The reason for that is that an external password rotation would be overwritten by the next terraform apply with the old value because Terraform sees the configuration drift and applies what it thinks is the desired configuration.

Instead we use the azurerm_keyvault_secret datasource for retrieving a secret from KeyVault. For that to work the secret must exist when the datasource is refreshed. If not the Terraform build will fail due to a ‘KeyVault Secret not found’ error.

There are several ways to create the secret.

One option is to create the secret in KeyVault manually prior running Terraform. This would be very bad from an automation perspective because secret creation (and deletion) should be completely automated.

We spend quite some time on this issue and came up with a different solution. We are having a null_resource with a local-exec provisioner which is run once and makes sure the secret is created in KeyVault:

resource "null_resource" "create-password" {
  provisioner "local-exec" {
    when    = create
    command = <<-EOF
  set -e  
  password=$(openssl rand -base64 32)
  
  # output none to NOT display the passwords in logs
  az keyvault secret set --vault-name "$KEYVAULT_NAME" --name "${SECRET_NAME}" --value "$password" --output none

EOF
    environment = {
      SECRET_NAME            = var.secret_name
      KEYVAULT_NAME          = var.key_vault.name
    }
  }
}

data "azurerm_key_vault_secret" "password" {
  depends_on   = [null_resource.create-password]
  key_vault_id = var.key_vault.id
  name         = var.secret_name
}

Then we use the normal azurerm_keyvault_secret datasource to use the secret. The datasource has a depends_on attribute to make sure the secret is created first!

During the first Terraform run you will see that the datasource is evaluated deferred because of the depends_on block. Following runs will evaluate the datasource during plan phase because the null_resource is already existing and not changed.

If the secrets value is changed in KeyVault the next Terraform execution will pick up the new value and following modules can change their configs accordingly. We have a small shell script which runs on a schedule and updates the secrets in KeyVault and triggers the Terraform pipeline afterwards.

To make sure the secret gets removed from KeyVault when it is no longer needed we use a destroy provisioner:

resource "null_resource" "delete-password" {
  #because Destroy-time provisioners and their connection configurations may only reference attributes of the related resource, via 'self', 'count.index', or 'each.key'.
  triggers = {
    name           = var.secret_name
    key_vault_name = var.key_vault.name
  }
  provisioner "local-exec" {
    when    = destroy
    command = <<EOF
  set -e
  if [ -z "$SECRET_NAME" ]; then
    echo "Secret name to delete is not set(\$SECRET_NAME)."
    exit 1
  fi
  if [ -z "$KEYVAULT_NAME" ]; then
    echo "KeyVaultNAME is not set(\$KEYVAULT_NAME).)"
    exit 1
  fi
  echo "Deleting secret ${SECRET_NAME} from vault $KEYVAULT_NAME"
  az keyvault secret delete --only-show-errors --vault-name "$KEYVAULT_NAME" --name "${SECRET_NAME}" || echo "Could not delete secret ${SECRET_NAME}"
EOF
  }
}

Hint: For this to work you must have the Azure CLI installed and be logged in with a user with KeyVault secret set and delete permission!

If you are using a KeyVault with soft delete enabled you should check in the create script if there is a deleted secret with the same name as you are trying to create. This can happen if a secret with that name was destroyed by Terraform and is now about to be recreated. If so you must recover the secret first before you can set a new value. For that add this at the beginning of the create secret script:

echo "Checking if the secret exists in DELETED state and recover them, otherwise we cannot use the name again"
deleted_secret_exists=$(az keyvault secret list-deleted --vault-name "$KEYVAULT_NAME" --query "contains([].id, 'https://$KEYVAULT_NAME.vault.azure.net/secrets/$SECRET_NAME')" -o tsv)
if [ $deleted_secret_exists == true ]; then
  echo "Deleted secret exists. Recovering them before setting new password values"
  az keyvault secret recover --only-show-errors --output none --vault-name $KEYVAULT_NAME --name "${SECRET_NAME}"
  sleep 2 # prevent 'is currently being recovered and cannot be re-created; retry later.'
fi

In this case the Azure CLI logged in user needs the secret recover permission as well.

# Final words

Sometimes there are constraints for what Terraform is made for. Especially date handling and more complex (dynamic) conditions can bring the HCL to its limits. Something you would solve in a programming language in a few lines can take in HCL very complex conditions an strange behaviour during execution time.

It is quite disappointing that the Azure KeyVault offers no option to generate a secret on-demand like it does for instance for certificates.

We hope that at some point in the future the KeyVault is able to auto create and rotate secrets. But until then we are quite happy with our solution.

Discovery of a new architectural pattern

This newly discovered architecture was too good (and funny) to be kept secret. We called our observation the “many-services architecture” it is a revolutionary(?) deviation from the microservice architecture.

The basic principle was already discovered during the rise of continuous deployment and is pretty simple:

“If it hurts, do it more frequently, and bring the pain forward.” [Source: 1]

For service design that means we don’t deploy only a few services but as many services as possible. (“The more the better!”)

Implementation

It is important to not worry about service boundaries, domains and other stuff which just confuses your head and is changing all the time anyway but think more like: “Start small, implement fast, test later!”

You did everything right when there is at least a handful of services per team member. All services need to be cleanly separated in different Git repositories in order to prevent merge conflicts.

In some agile handbooks this is called two-pizza teams. (Source: 2) Think of your services as pizza slices. Each team member owns at least as many services as there are slices. A tasty idea and easy to follow!

Communication and Integration

Integration patterns are always different to explain and stuff gets usually complicated when you are no longer alone. Thats the reason not to be shy from the beginning:

The services only communicate asynchronous (best option to start: self-hosted Kafka cluster!) and each service should at least be dependent to three other services (“all good comes in three”). If the last part cannot be fulfilled think about deleting your service, it doesn’t seem to fit very good in your application landscape!

The holy grail is if every service is calling every other service without running into cycles!

(If you do: Don’t worry, just implement some sleep(100) command in your code. It is asynchronous, nobody will notice anyway!)

Important for the service integration is not to rest on using REST! If the payload of service calls is human readably it can’t be fast. Use at least gRPC or something more fancy in order to squeeze out every millisecond during service calls in order to make up for the 10 seconds unoptimised database query you have, because nobody sees or reads database logs.

Polyglot languages

To leverage the full potential of the many-services architecture different languages should be used. This will be attractive for new employees because you have the perfect programming language for every job candidate!

If in doubt if you should use a certain language the rule of thumb is:

If it can be containerized it will work!

Security

Everybody is talking about DevSecOps and security questions must not fall short in the many-services architecture. ( ̶S̶e̶c̶u̶r̶i̶t̶y̶ -f̶i̶r̶s̶t̶ /security-somewhere in-the-top-10-of-important-things)

For authentication and authorization it is important to mix protocols. (“Security by obscurity!”) Use at least some OAuth2 and SAML to complicate lateral movement and make it not to easy to use your stuff (there is a reason we are all experts, right?).

Don’t rely on a single identity provider, the more identities the better. For fault tolerance rely on basic auth with application local users for all admin stuff. This way you can administer your applications even if the identity provider is offline and passwords can be recovered from the webserver logs by taking a look at the http headers!

Don’t get lost in the details of authentication and authorization, a single OAuth2 application client is fine for alle your services! That makes JWT validation easier anyway.

Deployment

Continuous Deployment got a lot of attention the recent years. Together with the DevOps culture it got even more important.

Nevertheless it is important for the many-services architecture to rely on battle-proved techniques. Since you have a lot of services the Ops people need to train deploying! Use CLI tools with imperative commands.

Don’t think about GitOps and all that complexity of deployment tooling. There are just too many tools around, who can really make an informed decision???

Important: Don’t spend too much time searching for CVEs and other weaknesses in your applications: What you don’t know cannot make you worry (too soon)! The later you find a problem the less likely is deployment will be stopped because nobody wants to miss a project milestone!

To secure your back you should distribute responsibilities between different teams in your organisation. This creates plausible deniability if the customer blames you for errors and problems.

Outlook

With all this great hints and techniques you should be busy for the near future.

But(!) there are already the next new pattern evolving at the horizon.

Java developers keep an eye on the new “one-deployment-per-class” pattern and the frontend people should start not too late with the “one-web component-per-button” implementation.

I hope our findings enlightened you a little bit as well. Feel free to add missing architectures and patterns in the comments!

Disclaimer: Don’t take this too serious ;-)

Sources and References:

[1] Jez Humble, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation

[2] Two-Pizza teams, Introduction to DevOps in AWS

OpenCost enables you to visualize and measure the costs produced by kubernetes workloads:

OpenCost is a vendor-neutral open source project for measuring and allocating infrastructure and container costs in real time. (source: OpenCost.io)

It is nice to see what costs a deployment or a persistent volume causes in the Kubernetes cluster. But apart from these compute and storage costs there might be cost outside your cluster which should be included in the cost report.

Examples for this type of costs can be PaaS instances like databases (e.g. AWS RDS or Azure PostgreSQL), container registries and other services which are not running inside your clusters.

In my use case this was from special interest because I am part of a platform team providing a product to development teams. The development team needs to know what cost they produce in order to be able to run their applications in a cost efficient manner.

**Sidenote**: During my tests it wasn't really clear to me what is part of the CNCF Project OpenCost and what is part of KubeCost (and furthermore what is a paid feature of KubeCost). Being a sandbox CNCF project I guess there is still some work left on the documentation side. Nevertheless it is already an awesome product.

Configuring the Azure cost export

While installing OpenCost inside an Azure Kubernetes Service (AKS) cluster I followed the documentation on the KubeCost Page for Azure Cloud Integration.

Having the cost export configured as documented results in this flow:

On Azure subscription level there is a cost export configured which creates a CSV file on a daily schedule. The file is exported to a blob container inside an Azure storage account. The export contains the monthly costs up to the current day.

This works probably fine for most environments. But if you are working in an environment which requires to use private endpoints for all your Azure resources their is a problem:

The CSV exports upload is coming from a IP address range located in the Azure Cloud Service Tag which contains nearly half the Azure-Internet ;-).

Using a private endpoint for the storage account where the costs exports should be exported to results in an error message because the blob container is not reachable. Configuring firewall rules for the storage account was no option in our case.

If the cost export cannot be delivered to us by the exporter then we have to get it ourselves. Going through the list of available Azure REST APIs we found the Generate Cost Report API . The API is generating exactly the CSV data structure the Azure Exporter does (which btw can be configured via API using the Export API). We just need to call the API on a daily basis ourselves.

Kubernetes cron job to the rescue

Since we already have a Kubernetes cluster the solution was quite obvious. We create a Kubernetes cron job with a daily schedule and create a shell script to call the Cost Report API.

Creating the cost report has these steps:

1. Generate Cost Report API is called to get the report for an Azure subscription
2. the APIs response has an header named location which contains the URL to check the reports creation result. We extract the header’s value
3. Call the Get Operation Results API (that is the link from above)
4. This time we get a JSON response containing a blobLink attribute with the URL to download the CSV file (located in some Azure managed storage account). The url contains a SAS token and can just be used to download the file.
5. Download the CSV file with above link
6. next we upload the report to our own storage account which gets crawled by the KubeCost installation (Gotcha: The path structure after uploading must match what the default Azure Export is creating. The schema is <exportname>/<time-from>-<time-to>/<exportname>-<random-uuid>.csv , e.g. test/20220801–20220831/test_74ddd963–1d30–427f-90fc-f0a960aa5a80.csv)

To get this working we need:

• Azure Service Principal with required permissions to create the cost report (“Cost Management Reader” role) for the target subscription. Credentials are stored in an Secret
• ConfigMap with the shell script to generate the report. This makes it easy to adapt the script to your environment
• CronJob definition for starting a Job instance on a daily schedule

First we define a secret which contains all the information to obtain the cost report. The Secret contains information about the service principal with the permissions to create the cost report, the subscription and tenant ids and the name of the storage account where the reports should be uploaded. This storage account must have a container named cost-analyzer.

Next we define a ConfigMap which contains the shell script to be executed by the CronJob. Alternatively you can build your custom OCI image and copy the script into the image. I like this variant because it allows easier editing and changing the shell script:

And last but not least the CronJob ties together the Secret and the ConfigMap. The schedule is to run a Kubernetes Job at 8 o clock (UTC) every day:

To test if your setup works you can trigger Job instances for your CronJob manually with

kubectl create job --from=cronjob/azure-cost-exporter manualrun1

If everything is configured correctly you shouldn’t see any error in the Pod executing the Job and in the target storage account a CSV file should appear containing your subscriptions Azure cost.

Restart the OpenCost Pod to enforce a scanning of the storage account. Otherwise every 3 hours the storage account is scanned for new cost reports.

I hoped that helped you to get the export working in a similar network constrained environment like ours ;-)