Hi! I’m Tim Strukov, QA Engineer at inDrive. Some time ago when I was updating tests for new endpoints, one of them behaved strangely. The service rejected a request, but on the next attempt it said the action had already been completed. In fact, nothing had happened.

The problem was with an idempotency key: the service reserved it before validating the request. If validation failed, the key remained reserved anyway. I prepared a fix, went through code review, and shipped the change. After the release, duplicate-key errors on the affected endpoints dropped by about 30%.

Previously, my work on a bug usually ended with a detailed report and retesting someone else’s fix. Now, with Claude, I can go through the whole process by myself: understand unfamiliar code, check a hypothesis against logs, prepare a patch and tests, submit the change for review, and take it through to release. Today, I independently ship an average of three to five small fixes and improvements a week.

I distinguish three levels of this approach based on my experience at inDrive, which I’ll describe below. It’s important to note that this is not a mandatory career path for every QA engineer, nor does it mean handing responsibility over to an agent. In this approach, AI helps investigate systems faster, but the engineer still checks the facts, assesses the risks, and takes responsibility for the result.

Level 1. A Bug Report That Lets a Developer Start Fixing Right Away

A typical bug report answers the question, “What is broken?” But the developer still has to find the cause, understand the related parts of the code, and choose a solution. To save developers time, I started adding the results of that investigation to my bug reports.

First, I ask an AI agent to trace the flow of data and find a possible cause. Then I check every conclusion in the repository: whether that piece of code actually runs in the scenario, how similar cases are handled elsewhere, and what other parts of the system the patch might affect. If a hypothesis isn’t confirmed, I don’t present it as a fact.

As a result, each ticket includes six sections:

1. The problem and the conditions that trigger it;
2. A specific proposed fix;
3. Acceptance criteria and a regression test;
4. Risks and solutions that look plausible at first but are actually wrong;
5. Impact on users;
6. References to files, lines of code, and related tickets.

This kind of report doesn’t guarantee that the proposed solution is correct. But it gives the developer a verified starting point instead of a task that still needs to be investigated from scratch.

Example: Passenger Requests That Never Reached the Driver

The inDrive order service has separate ride options for an assistive device, a service animal, and a wheelchair-accessible car. When the service built its response, all three values were mistakenly read from the same field: assistiveDevice.

As a result, the driver might not see a request for an accessible car if the passenger hadn’t selected an assistive device. And when the device option was selected, its description appeared under the other options as well. No tests covered this case.

Before the fix:

response.assistiveDevice = describe(order.assistiveDevice)
response.serviceAnimals = describe(order.assistiveDevice)
response.wheelchairCar = describe(order.assistiveDevice)

After the fix:

response.assistiveDevice = describe(order.assistiveDevice)
response.serviceAnimals = describe(order.serviceAnimals)
response.wheelchairCar = describe(order.wheelchairCar)

The cause was two copy-and-paste mistakes. In the ticket, I pointed to the exact location and specified the necessary changes and checks. The first available developer quickly made the fix, and the ticket was closed a couple of days later.

What matters here is not only the size of the patch but also the consequences of the bug: because of two incorrect field references, passengers who needed special ride arrangements might not get the service they required. That’s frustrating, and it had to be fixed fast.

What to Try in Your Team

Take a bug you can already reproduce:

1. Ask the agent to trace the flow of data from the incoming request to the system’s response.
2. Ask it to separate conclusions confirmed by the code from hypotheses, and to provide references to specific files and lines.
3. Check those findings yourself.
4. Then add the smallest possible fix, potential side effects, and a test that should stop failing after the patch to the ticket.

In another investigation, the agent helped find an endless recursive call in an error handler: for an unknown error, the function called itself with the same arguments. This could cause a stack overflow and crash the process. I explicitly noted in the ticket that the defect had been found by reading the code, not reproduced on a running instance. It’s essential to preserve that distinction.

Level 2. Prioritize Bugs Using Production Data

Even when you’ve found the cause, a question remains: how serious is the problem? How many requests or inDrive users does it affect? Can you confirm it not just from the code but also from how the system behaves?

For this, I analyze logs and metrics. The agent helps identify which events the relevant code writes, build queries, collect results across clusters, and group errors by cause. Where access to logging and metrics tools is available, MCP connectors are a convenient option.

But the results still need to be checked: the wrong filter or denominator can easily turn a convincing number into a false conclusion.

Example: Nearly 1.8 Million False Server Errors in One Week

While analyzing logs from the inDrive order service, I found a large number of 500 errors occurring during calls to the bidding service. To understand the causes, I used AI to build log queries and collect one week of statistics across 13 clusters.

In total, I found about 1.86 million such errors. Preliminary analysis showed that roughly 96.6% of them — about 1.79 million — were caused by clients cancelling requests.

The cause was in how errors were handled between services.

When a client closed the connection during a call to the bidding service, that service returned gRPC Canceled. The order service converted it into a generic 500 server error and wrote it to the logs.

Yet other parts of the service already handled these situations correctly: if the client closed the connection, the server did not try to send a response or log an error.

I checked whether these cancellations were caused by slow responses from the bidding service. They weren’t: calls completed in milliseconds. For example, the median duration of an unsuccessful bid retrieval call was 7.5 ms, compared with 5.6 ms for a successful one.

In other words, slow responses were not the problem. The client had already stopped waiting, while the server continued to count the cancelled request as its own failure.

As a result, monitoring showed errors where the server had not actually failed. Including related scenarios, client cancellations generated several million ERROR-level log entries in a week. This noise made it harder to distinguish real failures from expected system behavior.

Based on the investigation, I proposed changing the error handling:

1. If the client cancels a request during a call to the bidding service, don’t return a 500 error or write an ERROR log entry.
2. If the cancellation originates in the bidding service itself while the client is still waiting for a response, keep treating it as a server error.
3. Check how the change affects event consumers, and notify the owners of dashboards and alerts about the expected drop in error counts.

According to a preliminary estimate, this change could eliminate about 96% of the 500 errors in question. That’s the expected effect calculated from logs, not the result of a deployed fix. The findings also need to be reviewed with the bidding team.

This example shows why QA engineers should analyze production data. A high number of errors in monitoring doesn’t necessarily mean that users experienced the same number of failures. Before assigning priority and proposing a fix, you need to understand what exactly the system measures and how it classifies errors.

Another Example: Errors When Changing a Ride’s Status

While analyzing another scenario, I found that about 3% of requests to the ride-status transition endpoint failed over the course of a day. The logs showed that 90% of those errors occurred when the driver requested a status the ride was already in. And 55% of the errors involved a transition to “completed” for a ride that was already completed.

Timing data and traces pointed to repeated taps, stale screen state, and bursts of requests from an older ride screen. The next day, I repeated the analysis and got the same 90%.

We split the work into two tasks:

1. Return success if the ride is already in the requested status.
2. Add logging where we still lacked data — on the passenger side.

How to Verify an Agent’s Findings

Before putting metrics in a ticket, I check the time period the data covers, which clusters are included, what counts as an error, and what denominator is used to calculate each percentage. Then I check whether a spike can be explained by one instance or a rare scenario and, where possible, repeat the analysis over a different period.

It’s also important to distinguish three things:

1. An observation from the logs;
2. A hypothesis about the cause;
3. The expected effect of a fix.

They shouldn’t be presented together as one proven fact.

Level 3. Prepare a Fix, Pass Code Review, and Ship It

When the cause is clear, the risks are limited, and the change is small enough, a QA engineer can prepare the patch by himself instead of handing the task to a developer. Although AI helps write code and tests, this in no way means that a patch can be shipped without the usual engineering checks.

At inDrive, I started with simple changes reviewed by developers. Over time, I began taking on more substantial tasks. Now my process includes code analysis, testing, code review, the standard release process, and verifying the impact after release.

Example: An Idempotency Key Was Reserved Too Early

Let’s return to the bug from the beginning of the article. Some endpoints reserved the request key before validating its contents. If validation failed or subsequent processing ended unsuccessfully, the key stayed reserved. The user’s next attempt received a response saying an operation had been repeated, even though it hadn’t happened in the first place.

In simplified form, the original sequence looked like this:

reserve(key)          // the key is reserved before validation
validate(request)
process(request)

After the fix:

validate(request)
reserve(key)          // only a valid request reserves the key
try:
    process(request)
onFailure:
    release(key)      // the request can be retried after failure

For an operation that really has already completed, the conflict response remains. But the response now also includes an explicit reason that can be tracked in monitoring.

After the release, I checked the metrics. Duplicate-key errors on these endpoints were down by about 30%; the decrease held every day after the release and across every cluster with traffic. Median response time, the 99th percentile, and other errors showed no noticeable change. The additional load on the shared cache — about 0.01% — had been assessed before the merge and considered acceptable.

That result matters more than the fact that AI helped write a few lines of code: the fix reached production, and we could verify its expected effect.

A More Complex Example: A Bug in a Shared Authentication Library

One service was crashing at startup with an unclear error. At first, it looked like a broken response parser. By tracing execution with AI, I followed the issue beyond our service into the shared authentication library. When the auth service returned a plain-text error page, the client tried to parse it as JSON and lost the actual response status code.

While investigating the source of that error further, I found a more serious problem: a background job that was supposed to refresh keys regularly ran only once and then stopped. After the next key rotation, services using an older version of the library could keep using stale keys, causing token validation to fail.

We reproduced the scenario. The fix was reviewed by the owners of the shared library, and then a new version was released. This case shows why it’s important to investigate a root cause all the way through, including beyond the service where the error first appeared.

What I Don’t Delegate to AI

The agent is good at reading unfamiliar code, tracing connections between services, and drafting log queries, patches, and tests. But it can also confidently explain a cause that doesn’t actually exist.

That’s why I keep four decisions for myself:

1. What has been confirmed by evidence;
2. How broad the change should be;
3. Which tests are needed;
4. Whether the patch is ready to ship and what to check after release.

I also follow specific rules:

• In each ticket, I state what was reproduced and what was found only by reading code.
• If a conclusion turns out to be wrong, I add an explicit correction to the ticket history rather than silently replacing the text.
• Before merging, I measure the load on shared systems.
• I run automated tests in a test environment and separately note flaky tests and tests that were already failing.
• I don’t turn a small patch into a large refactor.
• I send changes to shared code to the owners of dependent components.

These aren’t special rules for AI. They’re ordinary engineering discipline, and they become essential when a QA engineer at inDrive starts changing production code.

Where to Start

1. You don’t need a big task for your first experiment. Choose a bug you filed this week and try only Level 1: use an agent to trace the implementation flow, verify the likely cause, and add code references, a proposed fix, risks, and acceptance criteria to the bug report.
2. Once you can verify the findings, add Level 2: take one metric from logs or monitoring and use it to confirm the scope of the problem. Don’t put a percentage in the ticket until you’ve checked exactly what it measures.
3. Move to Level 3 with a small, straightforward change — for example, removing a feature toggle that has been fully rolled out and is no longer needed. Prepare the patch and tests, pass code review by the code owners, and follow the usual release process. After release, make sure the system behaves as expected.

For me, the biggest change in QA work isn’t that an agent can write code. I no longer have to stop at reporting a bug: I can find the cause, confirm it with data, prepare a fix, and — when the risk is acceptable — take it all the way to production. Responsibility for quality still stays with the engineer.

Acknowledgements

My growth over the past few months has been possible thanks to the support of my colleagues: they review my code, ask questions, and help me grow and push me further. Thank you to the Driver Experience team and to all the colleagues at inDrive who share their practices and support this process.

From Bug Report to Production: How I Fix Bugs with AI was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Two and a half years later, our infrastructure had doubled in size — to around 530 Aurora MySQL and PostgreSQL clusters. At the same time, we had no physical servers left, and only eight database incidents occurred in the first half of 2026.

The team still consists of four DBAs and me. I count myself as half a person — I’ll explain why at the end. I’ll tell you how we got here, including what didn’t work out.

Where We Started

When I interviewed for this position in 2023, my future manager told me three things about the team.

First, when I started working at inDrive, there would be an enormous amount of manual work. Second, the workload was too high, and the team wasn’t handling it particularly well. And, most encouragingly, the team was going through a difficult period: the team lead and one of the senior engineers had either already left or were planning to leave.

My future manager also told me that some databases were on physical servers, others were in the cloud, and the migration was already underway. Apparently, that was his way of preparing me for what I saw in early 2024.

There were three engineers plus me, and another DBA was supposed to join us in a couple of months. Our on-premises infrastructure included around 20 physical servers running the company’s main databases, including the database supporting the monolith. In total, they stored around 20 TB of useful data.

In the cloud, the situation with Aurora MySQL was rather chaotic. This infrastructure had grown during the decomposition of the monolith. Parts of the monolith were being turned into microservices (often a separate one for each region), and their databases were moving to regional or global Aurora clusters.

As a result, we had 250–300 small clusters across 14 logical regions, or six AWS regions.

At the same time, we had to:

• Maintain databases on physical servers: upgrades, replica switchovers, periodic data reduction, and backups.
• Respond to monitoring alerts and incidents.
• Handle L1/L2 service requests.
• Create new clusters in the cloud and migrate data from on-premises infrastructure.
• Apply DDL migrations, which could be dangerous and take an endlessly long time.
• Optimize queries, of course.
• Deal with a constant stream of “Could you also do this?” requests coming simultaneously through Slack channels, direct messages, and Jira.

Processes? Well, let’s just say they existed. Incident Management and Change Management were only beginning to emerge and mostly lived in chats. Users had a bot for creating requests, but the form essentially consisted of the question “Physical servers or cloud?” and a free-text field. Meanwhile, a significant part of the work was still agreed upon in direct messages.

It’s worth sharing some numbers here so that all this doesn’t sound too abstract. In 2023, the team completed around 2,400 Jira tasks, approximately 2,200 of which were user requests.

Ten tasks per working day doesn’t sound too bad until you remember that migrating a huge database to the cloud and granting access to a single person were in the same queue. So our delivery times weren’t particularly good.

And in the first half of 2024, there were around 50 database-related production incidents. Many of them caused complete or partial unavailability of company services.

What We Decided to Change

One circumstance worked in our favor. The company had already officially decided to move away from physical servers, so nobody expected us to continue developing the on-premises database infrastructure.

That left us with two goals: get the team out of the endless stream of requests and survive the cloud migration without bringing the business down along the way. I can’t say we solved the problems sequentially, one after another. In real life, everything usually happens at the same time.

But looking back, all the work can be divided into three stages:

1. We made the work transparent.

2. Then we handed over tasks that didn’t require DBA expertise to other teams.

3. After that, we taught the infrastructure to handle more load without our constant involvement.

Step 1. Make the Work Transparent

Initially, our Jira was completely flat. Formally, we had a Kanban board, but in practice, it had quite a few characteristics of the waterfall model.

A high-priority project task, technical debt, an idea along the lines of “let’s write this down before we forget,” and a user request — everything was in the same stream.

Prioritizing was difficult. So was understanding what was happening in general. Almost every morning started with messages like: “Hi! Could you please tell me when you’ll be able to work on request #…?” And many of those requests had long since passed any reasonable deadline.

People wrote both to the DBA Slack channel and directly to engineers, quite logically assuming that if they contacted everyone, somebody would eventually help. The backlog regularly reached 150 tasks, around 100 of which were user requests.

Everyone had become used to it: requests accumulated, people reminded us about them in direct messages, everyone felt a little uncomfortable, but for some reason nobody considered what was happening a good enough reason to change anything.

The engineers already had a concept of capacity — an empirically determined number of tasks each person could work on simultaneously. Once you reached the limit, you didn’t take on a new task until one of the current ones was completed.

In addition to this, we divided the work into separate streams:

• Project work.
• Service requests.
• Everything else.

For the first time, we could see user requests separately from quarterly goals. A little later, we started grouping project tasks into epics, which made it easier to manage large initiatives and distribute work among engineers.

We also got colleagues into the habit of creating requests even if the work had already been discussed and completed in direct messages. At least for tracking purposes. Finally, we committed to SLAs. At first, they were quite abstract, but users finally had at least some certainty.

Did the number of requests decrease after that? Of course not. But now we could see them and prioritize them. And we stopped making people wait a month for a database deployment that actually required one hour of work.

However, we went a little too far in some areas. Two almost unnecessary streams appeared: “New Services,” exclusively for deploying new databases, and “Quick Tasks,” for requests that took a couple of minutes.

This year, the infrastructure teams, including ours, also stopped using the bot and switched to a regular Service Desk portal for creating requests. But the portal itself isn’t the main point here. What matters is that a request is no longer arbitrary text. It’s a proper task with a clear Definition of Ready and a defined SLA.

Step 2. Hand Over Work That Doesn’t Require a DBA

Transparency didn’t reduce the flow of tasks. So we looked at what DBAs were spending the most time on, even though the work itself didn’t require their expertise. Two things stood out: access management and database schema migrations.

Database Access

I think every DBA team spends time granting access. In 2024, our users accessed databases through Redash, and the system wasn’t particularly convenient. Only DBAs could administer it, and only one engineer from a neighboring team knew how to fix Redash when it broke.

Besides, Redash connected to all databases using a single shared account, so it was practically impossible to track the actions of a particular user. The Information Security team didn’t like that. And I completely understand why.

So, together with InfoSecurity, we started migrating user access to HashiCorp Boundary. We enabled activity auditing across the entire infrastructure — around 300 databases at the time — and prepared preconfigured roles.

Instead of using one shared account for everyone or a separate database account for each user, we create roles directly in the database and provide access to them through Boundary. Individual user permissions are then managed there.

Along the way, we solved approximately a million problems and wrote a kilometer of documentation together with our InfoSecurity colleagues. Thanks, guys. And eventually, users moved to the new system.

And since access through Boundary had become standard and no longer required DBA involvement, we handed over access approval and provisioning to the IAM team, which already manages most standardized access in the company.

By the way, we only provide read-only access.

Database Schema Migrations

Previously, DBAs applied every schema change across the entire infrastructure themselves. When I joined, the tools were already quite good: Argo Workflows + Liquibase, and sometimes pt-online-schema-change.

Migrations are versioned in Git, and their execution history is stored in the database. This makes it possible to reproduce the schema across all environments and regions where a service runs.

Argo provides additional control: if a service runs in ten regions, you can first apply a migration in a less heavily loaded region and, if everything goes well, roll it out to the remaining regions.

The problem was the execution itself. DDL can leave a database hanging for an indefinite number of hours. Migrations often take a long time and don’t always succeed on the first or even second attempt.

You start a migration and technically can work on something else, but in practice, you keep coming back to check how it’s going. Users weren’t particularly relaxed about this either.

With more than 300 databases and over a hundred constantly changing services, manually reviewing and applying every DDL request within a reasonable timeframe was practically impossible.

So, together with the architects, we developed a process we called app-flow. It allowed teams to apply migrations independently. Implementing it wasn’t easy, but the main idea was quite simple.

A service team can independently apply migrations to its databases through the same Argo Workflows if it has the necessary permissions and follows a few rules:

• Migrations are stored in the service repository.
• The migration has passed code review.
• One of the SQL advocates has approved its execution.

Initially, the SQL advocates were architects and DBAs. Over time, developers joined them. But even today, DBAs still help if something goes wrong or suggest a better way to solve a problem. We also kept responsibility for migrations of business-critical services.

How This Affected the Numbers

The decrease in service requests doesn’t mean our colleagues simply gave up on us. It’s a direct result of teams starting to apply migrations independently and access management no longer being a DBA responsibility.

And remember: over the same period, the infrastructure roughly doubled in size. The backlog also changed: today, it usually contains around five service requests (closer to ten right now, and, frankly, that already feels like a lot), around 50 project tasks, and 25 tasks in the “Everything Else” stream.

The queue almost halved, but more importantly, its composition changed. Previously, user requests made up more than half of the backlog. Now they account for roughly one-tenth.

Previously, we worked on projects only when we managed to deal with the incoming queue. Now project work accounts for around 60% of tasks.

Step 3. Teach the Infrastructure to Work with Less Involvement from Us

We directed the time we’d freed up toward the infrastructure itself. Operating databases in AWS seems simple until you have several hundred clusters.

Every cluster needs monitoring, backups, access permissions, and policies, and sometimes additional resources such as S3 buckets. Besides, clusters differ in criticality and workload.

You need to figure out how to execute queries across multiple regions for a single service, when Global Aurora is better suited than Cross-Region, when to use Aurora Serverless and when to use Provisioned, and how and when to upgrade the engine version. Write automation scripts, and so on — a million things to do.

Moving Away from Physical Servers

We migrated all databases to AWS and completely eliminated physical servers. Along with them, an enormous amount of maintenance work disappeared.

Infrastructure as Code

Most changes go through a repository + GitHub Actions + Terragrunt. Some operations still can’t be performed through code: promote, blue-green, switchover, sometimes urgent backup operations, or very large-scale but minor changes that we later reflect in the configuration.

However, most of the configuration is defined in code, which means code can be the single source of truth.

Elasticity

For now, this mainly applies to read replicas. Every Provisioned Aurora cluster has autoscaling that adds read replicas when read load increases — without human involvement.

To achieve this, we had to write our own Terragrunt module because the existing ones didn’t solve our problem. For write workloads, we built a separate autoscaling mechanism using Step Functions.

But AWS alone wasn’t enough here. Not every service can survive a database switchover without an error like:

MySQL server is running with the — read-only option…

Our services are written in Go, and the standard driver everyone uses has the parameter `rejectReadOnly=false` by default. So part of the solution had to be implemented on the service side.

Service Maturity Model Rules

SMM stands for Service Maturity Model. These are automated checks that monitor whether services comply with architectural and infrastructure requirements.

At our request, a new check was introduced. If a new service doesn’t use a specific version of the database connection library, with all the necessary parameters configured correctly, it receives a warning.

When the check becomes mandatory, such a service won’t be able to pass it and won’t be allowed to deploy.

Partitioning

Following our recommendations and those of the architects, most services eventually concluded that life is easier with partitions.

Usually, queries access the most recent one to three partitions rather than scanning the entire table. As a result, table size no longer has such a direct impact on response time.

Partitions also form the basis of our cold data management mechanism, which I’ll discuss below.

Change Management

It’s not perfect, but at least it exists now. All production work is announced in a dedicated channel. This is useful for reporting, and if something breaks badly, we can reconstruct the entire sequence of actions related to the affected service.

Besides, when you announce your work to more than 300 people, for some reason you start paying a little more attention to what you’re doing.

How This Affected the Number of Incidents

In the first half of 2024, we had around 50 database-related incidents. Some were caused by the actions of DBAs themselves, others by configuration errors, insufficient capacity of production AWS instances, or MySQL on physical servers, which were still actively used at the time.

In the first half of 2026, there were eight incidents. Only three of them were reported to us by people outside our team. Usually, engineers detect problems themselves through alerts and create an incident for the service team through our Incident Management process so that we can investigate the situation together.

Two incidents were serious:

1. In one, an entire region went down completely.

2. In the other, during a major global event, the read and write (RW) load of a core service in one of the regions increased fourfold in approximately 15 minutes.

By the time the incident was created, the DBAs were already provisioning a more powerful instance to switch the RW workload over to it.

What We Spent the Freed-Up Time On

With less routine work, we had time for things we’d never managed to get around to before:

1. DynamoDB support — probably one of the most interesting tasks we worked on in 2025. Today, we have around 260 tables, both global and local, with separate backups through AWS Backup. Some of them send data to the data warehouse through Kinesis.

2. Aurora PostgreSQL is used mainly for geoservices, where PostGIS is practically indispensable.

3. We built PCM — Partition Cooling Mechanism, a mechanism for cooling partitions.

Company policy says that permanently deleting data from the source is generally a bad idea: you never know when you might need it again. But if you never delete anything, sooner or later you’ll run out of storage space. PCM regularly moves old “cold” partitions to S3 buckets — in a controlled manner, with the ability to verify the result.

4. Several AWS Region Moves — moving our entire infrastructure from one AWS region to another.

I used to think that moving everything from one AWS region to another was an incredibly complex project involving data loss and enormous downtime. It turned out that with a good team, it’s mostly just an incredibly complex project requiring long and intensive preparation. And the actual downtime for users can be less than half an hour. I think we switched the last region in about ten minutes without losing a single byte. In total, we completed four such migrations, not counting the initial move to the cloud.

Some failures and events also made us think more seriously about BCP/DRP — ensuring infrastructure reliability and recovery under various failure scenarios.

5. And, of course, there’s everything else that can be automated:

• Does a user need custom automated backups?
• Does the Data Platform team want to regularly run heavy SELECT queries against the main clusters without affecting the main service?
• Do we want to set up scheduled password rotation for service and technical accounts?

Hooray, we finally have time for all of this.

What Didn’t Work

1. Additional Jira streams from the first step. More structure doesn’t always mean better.

2. RDS Proxy. We started using it and then abandoned it.

3. Redshift. Almost six months of preparation for a project that never took off. Priceless experience.

4. Write workload autoscaling still can’t be used everywhere because not all services can correctly handle database switchovers yet. That’s exactly why we need the SMM check.

Where We Are Now

• Around 530 Aurora MySQL and PostgreSQL clusters and approximately 1,100 instances.
• More than 1,000 TB of total storage capacity.
• Around 260 DynamoDB tables in various configurations.
• Zero physical servers.
• Eight incidents in six months instead of 50.
• Eight AWS regions and the same 14 logical regions, although their locations have changed slightly over time. We even managed to deploy one region and then roll it back. =)

For comparison: in January 2024, we had 20 physical servers with 20 TB of data, 250–300 Aurora MySQL clusters, two incidents per week, and an endless backlog.

If You Lead a DBA Team

Here are a few things I would tell myself at the beginning of 2024.

1. Divide incoming work into separate streams before trying to reduce its volume. It’s impossible to prioritize a queue where a major migration and an access request look exactly the same.

2. Find tasks that rarely require DBA expertise. Turn them into self-service processes with clear rules — reviews, approvers, preconfigured roles — and hand them over to those who already manage similar processes in the company.

3. Database-side autoscaling is only half the work. If services can’t survive a database switchover, fix the client library and enforce the requirements through automated checks.

4. Measure everything. It’s the numbers that show the team and everyone around it what has actually changed.

Where the 0.5 Person Comes From

Since spring 2024, the team has still consisted of four DBAs plus me. In mid-2024, I took on a second team — DMP (Data Management Platform), which is responsible for Kafka and its ecosystem, NATS, and ClickHouse.

Together, we became Stateful Systems, and the amount of attention I could personally devote to the DBA area objectively decreased. Hence my modest role as 0.5 of a person. But the team still grew. Just not in numbers.

The engineers turned out to be an incredibly strong team. That “difficult period” I was told about during my interview was clearly related not to the people, but to the conditions they found themselves in. And I’m not the only one who thinks so.

Every year, inDrive conducts an Engineering Satisfaction Index (ESI) survey, in which engineers evaluate their experience working with teams and tools within the Technology Division. This year, DBA/DMP confidently ranked among the top three.

The DBA role itself has also changed. Tasks have become larger and more complex, and solving them now requires coming up with new approaches and diving much deeper into service architecture.

Which, frankly, is much more interesting than granting access.

How We Scaled to 530 Aurora Clusters With a 4.5-Person DBA Team was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Authors: Alexey Sumin and Oleg Anisimov, QA Engineers at inDrive

A map can look noticeably different from one mobile E2E test run to another and still be completely correct. The camera may shift, the route may take a slightly different shape, or a pin may move a few pixels. A person can usually look at the screen and tell immediately whether something is wrong. A pixel-by-pixel comparison cannot.

This was a recurring problem for our GeoWay team, especially in tests involving the map layer. Many of our scenarios depend on routes, pickup and destination points, navigation, vehicle positions and other visual elements on the map. Appium is useful when you can check the presence or state of a specific UI element, but in our case it sees the map as a single layer rather than a set of elements with individual locators.

That means checking that a route exists is not enough. We also need to know whether it connects the right points, whether its geometry makes sense and whether the screen looks correct as a whole. We had been using a pixel comparer for this, but it produced a lot of noise. A change to a font, icon, margin or surrounding layout could slightly move the map and push the visual diff over the threshold even though nothing was actually broken.

Where pixel comparison starts to break down

Our original setup was fairly standard for visual regression testing. The project stores a golden screenshot, and during a test run the pixel comparer takes a new screenshot and calculates the difference between the two. If the diff stays below a configured threshold, the check passes.

This works well for stable interfaces. Maps are different. The camera may shift slightly, the zoom level can change, data outside the area we care about can affect the layout, and a routing provider may return different but equally valid geometry.

In one of our tests, for example, the comparer reported an 11.24% difference against a 5% threshold, although the route itself was correct.

There was no useful universal threshold we could set. Make it too strict and valid tests fail constantly. Make it too loose and you risk missing a real issue. So the test report kept filling up with red cases that someone still had to inspect manually.

The pixel comparer was useful as a first filter, but it often could not make the final decision.

Adding AI Judge

We kept the existing comparison logic and added another step only for ambiguous cases. We called it AI Judge.

The model is not involved in every test. Most checks still end with ordinary deterministic logic. AI Judge is called only when the pixel difference is too large for the normal threshold but still small enough that we consider the result worth analysing rather than rejecting outright.

The flow is:

1. Run the UI test and take a screenshot.
2. Compare it with the golden screenshot.
3. If the diff is within the normal threshold, pass the test.
4. If it is above that threshold, check a second limit configured for AI Judge.
5. If the diff is within that range, send the expected and actual screenshots to the model. If it is above the upper limit, fail the test immediately.
6. AI Judge checks the screenshots against the expectations defined for that specific test and returns pass or fail.

The limits are configurable per test. A relatively dynamic case might allow AI Judge to inspect differences of up to 40%, while a more stable screen can use a much lower boundary.

The important part is what we ask the model. We do not send a generic prompt such as “Are these screenshots similar?” Instead, each test describes what actually matters. For a route, that may mean checking that it is continuous, connects the right points and shows the expected icons.

This changes the task from visual similarity to a much narrower question: does the actual screen still satisfy the test expectations?

If it does, AI Judge can rescue the test. We mark those cases separately so we can track them and review a sample manually. The model also returns a confidence score, and we define the minimum confidence we are willing to accept. If it finds a problem or is not confident enough, the test stays red.

The goal here is not to make the dashboard greener. It is to remove failures that are clearly noise without hiding real defects.

What happened with the 11.24% diff

The earlier example is a good illustration. The routing provider returned slightly different geometry, so the pixel comparer rejected the screenshot at 11.24%. The route still connected the correct points and was functionally valid.

AI Judge checked the actual test expectations and passed it. Previously, a tester would have opened the report, looked at the screenshots and reached the same conclusion manually.

This is essentially the role we wanted AI Judge to play. It does not replace Appium or visual regression testing. It handles a narrow class of cases where the deterministic check can see that something changed but cannot tell whether the change matters.

What changed in real runs

We compared two recent nightly runs, one without AI Judge and one with it.

The number of failing tests went from 47 to 8 in those two runs. On iOS, failures dropped from 15 to 4; on Android, from 32 to 4.

We do not treat this as a permanent performance metric. The composition and total number of tests change between runs, so a single comparison can be misleading. What matters more is what disappeared from the report: many of the cases removed by AI Judge were tests that previously reached manual review and were then confirmed to be valid anyway.

The result is less noise. A red test is now more likely to be something worth opening.

The bigger benefit: automating tests we used to run manually

Reducing false failures was useful, but it was not the main reason we built this.

Navigation scenarios were much more interesting. Because the route comes from an external provider, its exact geometry cannot be fixed ahead of time. A completely correct route may still differ significantly from the baseline screenshot, which makes these tests awkward to automate with a normal pixel comparer.

Before AI Judge, some of these cases simply were not worth automating.

In one regression suite of around 120 test cases, all 120 previously required manual execution. With the new approach, we were able to automate roughly 100 of them, leaving about 20 for manual testing. The team estimates that this saves around two hours of manual work per regression cycle, although the exact number varies by run.

For us, this is the strongest argument for the approach. AI Judge is not just cleaning up reports. It makes automation practical for scenarios where screenshot-based testing had previously been too unstable.

The model cost turned out to be almost irrelevant

We initially used GPT-5.4-mini and later moved AI Judge to GPT-5.6 Luna. In our tests, Luna performed better at analysing the screenshots while remaining cheap enough that model cost was not really a factor in the decision.

In recent runs, AI Judge processed 155,929 input tokens and 16,599 output tokens for iOS, and 83,187 input tokens and 30,118 output tokens for Android. Based on the pricing we used for the calculation in the project, that came to approximately $0.051 for iOS and $0.053 for Android, or around $0.104 in total.

Usage changes from run to run, so we do not treat those numbers as a fixed average. The useful conclusion is simpler: one regression run costs us a few cents, while manually checking noisy failures costs considerably more in engineering time.

We also track token usage directly in the test report so we can see how that changes as the test suite grows.

Prompt quality mattered more than we expected

Once a model can change the final status of a test, false passes become the obvious risk. During the first month we therefore kept manual review in place and regularly checked AI Judge decisions.

We found one questionable green result during that period. It was not an actual product bug, but according to our own expectations the model should have been stricter.

What we saw more often was a different problem: the prompt itself was incomplete.

If we forgot to describe something important, the model could make a reasonable decision based on the information we had given it. The mistake was not necessarily in the image analysis. We had simply left part of the specification out.

That changed how we think about prompts. They are now effectively part of the test code. If the route must connect points A and B, that needs to be stated. If colour matters, we need to say so. If an icon should be in a particular state, that condition belongs in the expectations as well.

We now review these prompts much like regular assertions. In practice, incomplete expectations have caused us more trouble than systematic image-recognition errors.

Automating reference screenshot updates as well

There is another piece of automation in the same nightly screenshot workflow, although technically it is separate from AI Judge.

AI Judge decides whether a test result is valid. A second workflow helps us deal with reference screenshots when they genuinely need to be updated. Alexey worked on AI Judge, while Oleg implemented this part.

Previously, someone had to find the right image in a large report, download it, add it to the repository and open a pull request manually.

Now, after every nightly run, an n8n workflow sends the results to Slack. The message includes the Android and iOS failures, the reason for each failure, a link to the report and the relevant screenshots.

If a screenshot should become the new reference, the tester selects it and clicks Update. The workflow then creates a GitHub pull request with the selected images automatically.

The two automations solve different problems around the same workflow: AI Judge helps us decide whether a visual difference is actually a bug, while the bot removes some of the repetitive work involved in maintaining reference screenshots.

What we learned

A few rules emerged after running this in the regression pipeline for several weeks.

We still prefer deterministic checks whenever they are enough. There is no reason to call a model if the pixel comparer can already give us a clear answer. AI is useful only in the ambiguous middle ground where understanding the content of the screen matters.

Keeping the task narrow also helps. AI Judge does not explore the product, generate the test or decide how the application should behave. It evaluates one known state against a set of explicit expectations. That makes its decisions easier to understand and easier to review.

We are also conservative with confidence. If the model is not confident enough, the test stays red. And even after launch, we continue reviewing real regression runs rather than assuming that a good result on a test dataset is enough.

What’s next

The problem we solved is not unique to GeoWay. Any mobile team working with maps, canvas-based interfaces, dynamic content or third-party visual components can run into the same limitation: the screen is valid, but it differs too much from the reference image for simple pixel comparison.

We are now looking at whether AI Judge can become a reusable module and have already discussed the approach with other inDrive teams working on geo-related functionality. There are also smaller improvements we can make around the workflow. For example, the nightly report could distinguish a brand-new failure from one that has been appearing for days. AI could eventually suggest reference screenshot updates and prepare the pull request, leaving the final approval to a person.

The key point is that we did not rebuild the testing stack around AI. Appium, the pixel comparer and the existing thresholds are still there. We added the model only at the point where deterministic checks stop being able to tell us whether a visual difference actually matters.

For our team, that turned out to be a small but useful addition to the pipeline. The model is not deciding how the application should work. It is helping us answer a much more practical question:

Is this visual difference acceptable, or is it a bug?

Special thanks to Karim Azab, GeoWay Tech Lead, for supporting the team while we worked on this automation, and to Vlad Ryzhkov. AI Judge grew out of Vlad’s earlier work on a workflow that analysed localization from screenshots; we adapted the same general idea for dynamic geo scenarios.

How inDrive Automated UI Testing for Dynamic Screens with AI was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

We replaced that file with a dependency graph and added parallel apply to our GitHub Actions pipeline. Over three and a half months, 355 parallel rollouts took 64 hours in total. Applying the same modules serially would have taken an estimated 171 hours: a 2.7× aggregate speedup.

The result depended heavily on the size of the change. Small PRs often became slower, and getting the dependency graph into a state we could trust took work before we could run anything in parallel.

How our Terraform CI pipeline used order.txt

Our repository holds configuration for networks, databases, Kubernetes clusters, IAM roles, and other AWS resources. Engineers propose changes through pull requests. GitHub Actions posts a Terraform plan for review; after approval, an engineer comments tg apply, and a runner applies the changes and reports the result in the PR.

For a PR touching several modules, CI needs to determine their apply order. A database that uses a new VPC must wait for that VPC to exist. Two unrelated resources in different regions may be able to run together.

The old pipeline resolved the order using a manually maintained list of glob patterns:

core/terragrunt.hcl
vpc/terragrunt.hcl
tgw/terragrunt.hcl
vpn/*/access_policies/terragrunt.hcl
vpc-endpoi*/terragrunt.hcl
sns/*/terragrunt.hcl
...

Position in the file determined execution order. New module types needed an entry, and missing entries could cause problems during a production apply. The vpc-endpoi* pattern covered inconsistent directory names. The file had accumulated quite a few details like that over the years.

It also gave us no way to express which changes could run independently. Once CI sorted the selected modules, it applied them one at a time.

What serial Terragrunt applies cost us

To measure the delay, I joined approximately 29,000 PR comments with 10,800 apply workflow runs, starting with records from November 2025. The six-month baseline contained 2,036 apply sessions.

The total describes time spent waiting for CI across those sessions. We did not measure how much of it prevented engineers from doing other work.

The opportunity was in PRs with independent changes. A rollout touching ten unrelated modules still executed ten consecutive applies. The ordering file put them in a sequence, but their dependencies did not require one.

Building and validating the Terragrunt dependency graph

Terragrunt configurations already describe dependencies in dependency blocks. We used those declarations to build a graph, then applied Kahn’s topological sorting algorithm to calculate the execution order for the selected modules.

The first problem was incomplete declarations. Some modules read outputs from other modules without a corresponding dependency block. Those relationships had to be made explicit before the computed order could be trusted.

We scanned the repository for undeclared references and fixed the configurations we found. We then added an Open Policy Agent check to CI to reject missing declarations, broken dependency paths, and cycles. A PR must pass that check before it can merge.

We archived order.txt once this validation was enforced. Without the check, later changes could have reintroduced the same missing dependencies we had just removed.

Grouping modules into dependency layers

The graph lets us divide a PR’s modules into layers. Layer 0 contains modules with no dependencies on other modules selected for that apply. Each later layer contains modules whose prerequisites in the selected set belong to earlier layers.

Modules within a layer can run concurrently. The next layer starts only after the current one finishes successfully.

This simplified example shows six modules across three layers. Arrows point from a prerequisite to a module that depends on it:

A, B, and C can start together. Once all three finish, D and E can run together. F runs last. This reduces six consecutive module applies to three stages, though elapsed time still depends on module duration, available runners, and the concurrency limit.

Running parallel Terragrunt applies in GitHub Actions

GitHub Actions supports matrices populated from an earlier job’s output. We use a matrix to run one job per module within a layer. The layer jobs themselves and their dependency chain are declared in the workflow YAML. See GitHub’s documentation on matrices defined from job outputs.

Our workflow predefines ten apply-layer-N jobs. Conditions skip the layers a PR does not need. If a change requires more than ten layers, the parallel workflow rejects it with instructions to split the PR or use serial apply.

Within each layer, strategy.max-parallel: 20 caps concurrent module jobs at twenty. The limit is configurable because AWS API limits and runner availability affect how much concurrency is useful.

Between layers, needs: requires the previous layer to complete successfully. If a layer fails, later layers do not start. This follows GitHub Actions’ job dependency behavior.

The barrier applies to the whole layer. A module in the next layer may wait for an unrelated slow module in the current one. That is a tradeoff of the implementation: we keep the execution order in the workflow’s job dependencies, at the cost of some potential concurrency.

Measuring the speedup against a serial baseline

We estimated serial execution time separately for each parallel rollout. For each run, we summed its modules’ measured apply-step durations and added the fixed overhead of the serial pipeline. We measured that overhead across 87 serial runs; it was approximately 26 seconds.

Using the same run’s module durations keeps the comparison tied to the work that was actually performed. It remains an estimate: running those modules sequentially could change their individual durations.

The results after three and a half months were:

The 2.7× figure is the ratio of total estimated serial duration to total measured parallel duration. Individual PRs varied considerably.

The 107-hour reduction describes elapsed rollout time across the measured runs. It does not establish an equivalent saving in engineering effort or runner costs.

How PR size affected parallel apply performance

The largest gains came from changes with many independent modules. One 64-module rollout finished in 31 minutes, against an estimated serial duration of 4 hours 40 minutes. A 206-module rollout, with every module in one dependency layer, finished in 44 minutes against an estimated 2 hours 53 minutes.

PR size alone does not determine the result. A wide graph gives the pipeline more work it can run concurrently. A long chain of dependent modules leaves less room for improvement.

Why serial apply is still the default

Of the 355 parallel rollouts, 124 took longer than their estimated serial baseline. Most were small changes, where dependency discovery and preflight work cost more time than parallel execution saved.

We therefore kept parallel apply opt-in. Our internal guidance is to use serial apply for fewer than four modules. Changes with four to seven modules were roughly break-even in this dataset; larger changes were more likely to benefit.

Keeping the serial path also gives engineers an option when a PR exceeds the parallel workflow’s ten-layer limit.

Why we kept orchestration in GitHub Actions

Terragrunt has its own dependency-aware run queue, including concurrent execution and filtering. Its current CLI also supports selection based on Git changes. The command previously known as run-all is now covered by run — all in the redesigned CLI.

Our implementation kept module selection from the PR diff, approvals, runners, and reporting within the existing GitHub Actions workflow. That let us add parallel execution to a process engineers were already using.

An external orchestrator such as Atlantis or Spacelift was another option. We already had the surrounding CI workflow in place and chose to keep scheduling there. Teams building a new pipeline should compare that choice with Terragrunt’s native run queue and the capabilities of their existing tooling.

What we would do first in another monorepo

The dependency audit would come first again. An incomplete graph can produce the wrong apply order, and enabling concurrency makes that a production concern. Automated validation needs to stay in CI after the initial cleanup.

We would also measure small PRs separately from large ones. The aggregate speedup was useful, but the 124 slower runs determined our default. For our workload, parallel apply is worthwhile when a change contains enough independent work to offset its startup cost. Serial apply still handles the small changes well.

By Alexander Plotnikov, Staff Infrastructure Engineer, inDrive.

Terragrunt Parallel Apply in GitHub Actions: 2.7× Faster was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

The confession: Google Cloud was outside our Terraform CI/CD

For years, our infrastructure CI was single-cloud — not by decision, but by construction.

The pipeline that plans and applies every Terraform change in our monorepo filtered changed files through an allowlist of directory prefixes. That allowlist simply never learned that gcp/ existed. The apply-on-merge workflow went further and explicitly excluded Google Cloud paths. There was no Google Cloud authentication in CI either: the generated provider block was bare, with no project and no credentials.

The result was predictable but easy to overlook: around 97 Terragrunt units across approximately 25 Google Cloud projects were being applied from engineers’ laptops, using whatever credentials happened to be available locally. There was no plan gate, no policy check, and no central audit trail.

Nobody chose this operating model. It accreted while most of our attention went to the much larger AWS estate.

Fixing it sounded simple: add Google Cloud authentication, then flip the path filter. Authentication was not what got us.

Three constraints for a keyless AWS-to-GCP pipeline

Before touching the pipeline, we deliberately constrained the solution:

1. Terraform state stays in Amazon S3. All our Google Cloud state already lived there. Moving it to Google Cloud Storage during the same project would have doubled the migration risk without creating user-visible value.

2. CI runners stay in AWS. We operate an autoscaled GitHub Actions runner fleet on Kubernetes in AWS. A second runner fleet in Google Cloud would become another platform to patch, scale, monitor, and secure indefinitely.

3. No service-account keys. Exported JSON keys are the default answer to “CI needs Google Cloud access,” but they introduce long-lived credentials that must be stored, rotated, and protected. Our organization policy already blocked key creation, and we were not going to request an exception.

Together, these constraints meant that an AWS-hosted runner had to call Google Cloud APIs without stored keys while continuing to read and write Terraform state in S3. The control plane crossed clouds in one direction; the state backend stayed in the other.

Choosing Google Cloud Workload Identity Federation

Google Cloud’s answer to keyless cross-cloud authentication is Workload Identity Federation (WIF). Google Cloud Security Token Service trusts an external AWS role identity and exchanges it for short-lived Google Cloud credentials. No persistent secret is stored anywhere in the chain.

WIF still leaves an important architecture choice:

• Option A — direct resource access. Grant Google Cloud IAM roles directly to the federated AWS principal. No service account is involved.
• Option B — WIF with service account impersonation. Allow the federated principal to do one thing: impersonate a purpose-scoped service account. The service account holds the actual roles on target projects.

Instead of debating the options in a design document, we built both end to end against real projects with production-shaped trust conditions.

Then we attacked our own configuration. An AWS role from the wrong account was rejected during token exchange. A role from the correct account but with the wrong role name was rejected by the trust condition. Each identity worked only through its intended path. Nothing failed open.

Why service account impersonation won

Option B won clearly for four reasons:

• Lower identity-coupling cost. Our AWS role ARN appears in two places instead of being woven through IAM policies in approximately 25 Google Cloud projects. When AWS-side identities change, the blast radius is two configuration lines.
• Broader API compatibility. A service account is an ordinary Google Cloud IAM member accepted by every API. Federated principals are not universally equivalent, and our environment is BigQuery-heavy, where direct-principal support was not yet proven for our use cases.
• Mature observability and operations. Service accounts come with familiar usage metrics, last-authenticated data, impersonation alerts, and a kill switch that every on-call engineer already understands.
• A purpose-scoped authorization boundary. The federated AWS principal receives only the ability to impersonate the CI service account; project permissions remain attached to a Google Cloud-native identity.

Measure revocation before production needs it

We did not assume the kill switch worked as expected. We measured it.

• After disabling the service account, requests for new tokens failed within approximately one second.
• Already-issued tokens were rejected within one to two minutes.
• Re-enabling the service account restored the flow within seconds.

The test also exposed a constraint that now shapes daily operations: our organization policy caps impersonated-token lifetime at one hour. Any single Terraform apply must fit inside that window. That is the kind of limitation you want to discover in a test harness, not when a production apply dies at minute 61.

How the keyless AWS-to-GCP authentication flow works

At runtime, the AWS-hosted runner uses its AWS identity to obtain short-lived Google Cloud credentials:

1. The runner obtains temporary AWS credentials for the approved IAM role.

2. The Google authentication library creates a signed AWS request that represents that role identity.

3. Google Cloud Security Token Service validates the configured account, role, audience, and attribute conditions.

4. The federated principal receives permission to impersonate the purpose-scoped Google Cloud service account.

5. The Terraform Google provider uses the resulting short-lived access token to call Google Cloud APIs.

6. Terraform continues to use Amazon S3 as its remote state backend through the runner’s AWS permissions.

Two implementation details the documentation undersells

First, the Google authentication library’s AWS flow does not understand Kubernetes-style web-identity credentials directly in our runner setup. We had to materialize standard environment credentials before starting Terraform. Without that step, the library silently spent several minutes probing a metadata endpoint that did not exist.

Second, gcloud is not required on the runners. The Terraform Google provider can perform the Workload Identity Federation exchange natively from a small external-account JSON configuration file. The file describes the federation flow but contains no private key and no reusable secret.

What the security review added

We took the service-account-impersonation prototype to our security team with working evidence rather than slides:

• an audit-log event showing the raw AWS identity and the CI session name;
• an impersonation event naming the Google Cloud service account;
• negative tests for the wrong AWS account and wrong IAM role;
• measured revocation and recovery timings.

That made the decision meeting short.

Security had one hard requirement that we had not planned: allowlist the egress IP addresses of the runner fleet’s AWS NAT gateways on the Google Cloud side. Federation already restricts who can authenticate; the IP allowlist also restricts where authentication can originate.

The control is inexpensive and provides useful defense in depth. It also makes the NAT gateway IP addresses part of the trust surface. Those addresses now have to be documented, monitored, and synchronized whenever the network changes. For us, that was a fair trade.

The hardest PoC debugging happened in AWS, not Google Cloud

The most difficult proof-of-concept problem was not on the Google Cloud side at all.

Our PoC role lived in a different AWS account from the S3 state bucket. Cross-account S3 access requires three permission layers to agree: the role’s identity policy, the bucket policy, and an exemption in a deny-list. Meanwhile, S3 can return 403 in situations where the underlying problem behaves like a missing object.

Production runners use the same AWS account as the state bucket, so this problem was limited to the PoC. It still consumed most of the debugging budget.

The hidden 80%: migrating 131 Terraform states

Our original estimate was 10–15 engineer-days for authentication, path filtering, and Google Cloud policy weights. Halfway through the project, we opened the S3 state bucket to generate the backend configuration and found the real scope.

Years of laptop-driven applies had scattered state objects across inconsistent legacy prefixes. Some keys did not match the current repository paths. Some were orphans from deleted experiments. Others were near-duplicates. The CI pipeline could not safely go live against that layout.

The middle of the project therefore became a full Terraform state migration:

• 131 state objects re-keyed into a clean and predictable structure;
• every Terragrunt unit re-pointed to its new backend key;
• the repository and state bucket verified unit by unit;
• zero Terraform state loss.

The pleasant counter-surprise was the dependency-ordering engine. Everyone assumed that the logic deciding which units applied before others was AWS-specific. It turned out to be cloud-agnostic and needed validation, not the rewrite we had budgeted.

Rolling out Google Cloud without breaking the AWS pipeline

The final risk was not Google Cloud. It was regressing the AWS Terraform pipeline that dozens of engineers rely on every day.

Three rollout habits kept the change safe:

1. Canary the policy model. Our policy gate scores each Terraform plan by risk. We added weights for 45 Google Cloud resource types, but first ran them against synthetic Terragrunt units so the first real Google Cloud pull requests were not blocked by poorly tuned thresholds.

2. Test every workflow leg. Plan, apply, and destroy flows — both comment-driven and merge-driven — were proven in a disposable harness project before we enabled the Google Cloud path filter.

3. Run an explicit AWS no-regression check. We compared the existing AWS behavior before and after the filter change. The result was zero regressions.

The unglamorous ending

Two practices attracted little attention but are the first things we would repeat.

First, we wrote the revocation runbook before go-live: exact commands, measured timings, and named owners. Access that cannot be revoked under pressure is not access; it is liability.

Second, the PoC received its own teardown ticket. We removed harness projects, trust configuration, temporary IAM roles, and bucket-policy entries, while preserving the findings in an architecture decision record.

One last gotcha: deleted Workload Identity Federation pools remain soft-deleted for approximately 30 days. Even teardown has a tail that must be tracked to completion.

Key takeaways for multi-cloud Terraform CI/CD

• Hidden scope lives in the boring parts. Authentication was the interesting problem; the Terraform state bucket was the actual project. Audit the current state before estimating.
• Bring security a working prototype, not a slide deck. Audit evidence and negative tests shorten the decision cycle — and security may still add one requirement you did not anticipate.
• Prefer short-lived federated credentials over service-account keys. Workload Identity Federation reduces secret-management risk and creates clearer audit trails.
• Test revocation, token lifetime, and recovery as product behavior. Do not leave them as assumptions.
• A PoC is not complete until it is deleted. Prototypes with real cloud access cannot retire in place.

By Alexander Plotnikov, Staff Infrastructure Engineer, inDrive.

Zero keys, two clouds: how our AWS-hosted Terraform CI/CD deploys to Google Cloud was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

A TabBar looked perfect with four tabs and broke with five. Here is how we traced a one-pixel seam to edge-aligned drawing and made the component robust.

At inDrive, a design-system component is only finished when it behaves correctly in every supported state.

While building a TabBar for our Android design system, we found a case that looked perfect with four tabs and failed with five: thin dark lines appeared on both outer edges of a white vertical gradient.

The fix took only a few lines. Understanding why it worked was more valuable.

Four tabs passed. Five tabs found the bug.

The initial implementation

The component needed a gradient behind its content. We used Modifier.drawBehind because it gives direct access to the component’s DrawScope and lets us paint across the full drawing area.

A regular background modifier did not fit the modifier order and internal padding used by this component.

Column(
    modifier = modifier
        .drawBehind {
            drawRect(
                brush = Brush.verticalGradient(
                    colors = colors,
                ),
            )
        },
) {
    // TabBar content
}

One extra tab exposed the edge case

The TabBar uses weighted rows to distribute its items.

Adding a fifth tab changes the width allocated to every item and also changes the geometry used during animation. In that configuration, a dark seam became visible on both the left and right edges of the component.

Why a one-pixel seam can appear

When drawRect is called without an explicit size, Compose uses the size of the current drawing scope:

fun drawRect(
    brush: Brush,
    topLeft: Offset = Offset.Zero,
    size: Size = this.size.offsetSize(topLeft),
    // ...
)

Compose drawing APIs work in pixels and represent drawing coordinates with Float values. The Size type itself stores floating-point width and height values:

value class Size(val packedValue: Long) {
    val width: Float
        get() = unpackFloat1(packedValue)

    val height: Float
        get() = unpackFloat2(packedValue)
}

Layout measurement often starts from whole pixels, but animations, transforms, interpolation, and geometry calculations can still place visual edges on fractional pixel boundaries.

If a rectangle is drawn exactly to the visible boundary, rasterization and clipping may leave the final column of pixels only partially covered. The underlying background can then appear as a thin line.

This does not mean that floating-point arithmetic or drawBehind is generally broken. It is a fragile edge condition that became visible only after the fifth tab changed the component’s geometry.

The fix: overscan, then clip

The gradient changes only along the vertical axis. Extending it slightly beyond the left and right edges therefore has no visual cost.

We can draw a small horizontal overscan and clip the result to the component bounds:

Column(
    modifier = modifier
        .clipToBounds()
        .drawBehind {
            val overscan = 1.dp.toPx()

            drawRect(
                brush = Brush.verticalGradient(
                    colors = colors,
                ),
                topLeft = Offset(
                    x = -overscan,
                    y = 0f,
                ),
                size = Size(
                    width = size.width + overscan * 2,
                    height = size.height,
                ),
            )
        },
) {
    // TabBar content
}

The extra pixels absorb small edge-alignment errors, while clipping keeps the final output inside the component.

The visual result stays the same, but the drawing is no longer dependent on perfect coverage at the exact boundary.

A production note on caching

If the brush or other drawing objects are recreated frequently, drawWithCache can cache them until the drawing area or observed state changes.

For a simple component, the difference may be small. However, the cached version is worth considering when the draw block allocates objects on every frame.

What this taught us

Test the smallest and largest supported states. The default state is rarely enough for reusable UI components.

Include awkward geometry. Odd screen widths, different densities, weighted children, and animated transitions can reveal seams that never appear in static previews.

Build a complete design-system showroom. Every state, item count, theme, direction, and interaction should be easy to inspect before release.

Prefer robust visual bounds. For backgrounds that are invariant along one axis, a small overscan combined with clipping is often safer than drawing exactly to the edge.

Treat platform abstractions as implementations, not magic. Reading the source is often the fastest way to replace assumptions with evidence.

The wider engineering lesson

The scale is obviously incomparable, but the same category of lesson appears in much larger systems.

The Ariane 501 investigation traced the launch failure to specification and software-design errors in an inertial reference system, combined with testing that did not adequately represent the real flight domain.

Small assumptions become dangerous when the operating range changes.

In our case, the consequence was two dark pixels — not a failed launch. But the habit is the same: test the real range of states, question hidden assumptions, and make visual correctness resilient rather than accidental.

Pixel-perfect is not a screenshot. It is a component that remains correct when its inputs change.

Conclusion

A fifth tab exposed a one-pixel rendering seam that the default component state had hidden.

The solution was straightforward: extend a vertically invariant gradient slightly beyond the horizontal bounds and clip it back to the component.

The more important outcome was a stronger testing rule for our design system: every reusable component must be reviewed across its full state space, including the combinations that seem least likely to fail.

That is how we keep UI quality predictable at inDrive.

References

1. Android Developers — Graphics modifiers in Jetpack Compose
   https://developer.android.com/develop/ui/compose/graphics/draw/modifiers
2. Android Developers — Graphics in Compose and DrawScope
   https://developer.android.com/develop/ui/compose/graphics/draw/overview
3. Android Developers — DrawScope API reference
   https://developer.android.com/reference/kotlin/androidx/compose/ui/graphics/drawscope/DrawScope
4. Android Open Source Project — Size.kt
   https://android.googlesource.com/platform/frameworks/support/+/f2e05c341382db64d127118a13451dcaa554b702/compose/ui/ui-geometry/src/commonMain/kotlin/androidx/compose/ui/geometry/Size.kt
5. European Space Agency — Ariane 501 Board of Inquiry report
   https://sci.esa.int/s/wRdpjv8

Fixing One-Pixel Edge Artifacts in Jetpack Compose drawBehind was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

In January 2023, we ran GPT-3.5 through the same Go and SQL interview blocks we were using to hire senior engineers. We did not create a separate benchmark for the model or simplify the questions. By our internal rubric, it crossed the senior hiring threshold on both blocks.

That did not mean GPT-3.5 could do the job of a senior engineer. We did not test it on system design, navigating a large codebase, working through an ambiguous production problem, or taking responsibility for a decision. Still, the result exposed something uncomfortable: part of what we treated as senior-level signal was pattern recognition that a general-purpose model could already reproduce.

At the time, I dismissed the result as a workaround. It felt a little like using Google Translate instead of learning a language: useful, but not a real capability.

I was wrong about what the workaround meant.

Today, effective use of an LLM is part of the engineering baseline. The skill is not getting the model to produce an answer. The skill is defining the problem well, recognizing when the answer is weak, and remaining responsible for what happens when the result reaches production.

That is the important shift. AI does not only make engineers faster. It moves the boundary between execution and judgment.

I lead Tech Platform at inDrive, a global ride-hailing and services company operating in more than 40 countries. Our group is responsible for developer tooling, shared infrastructure, CI/CD, and engineering practices across product teams.

We do not build the product. We build the environment in which the product gets built.

By platform, I mean more than infrastructure and deployment pipelines. It includes the tools, standards, context, and feedback loops that shape how engineering work is done. As AI changes what individual engineers can produce, that environment has to change with them.

Speed matters, but speed is not the goal. The goal is to ship more of the right product without lowering the bar on quality, security, or accountability.

Here is how we are approaching that work.

When a Senior Signal Is No Longer Enough

The GPT-3.5 experiment did not show that an LLM was equivalent to a senior engineer. It showed that some of the signals in our interview process were no longer sufficient.

The model performed well when the problem was already structured and the expected output was clear. It struggled when it had to define the problem before solving it, work through incomplete context, or make an original judgment across several conflicting constraints.

That distinction changed how we think about technical interviews.

Our Go interviews now run in CoderPad with the built-in AI chat available to both the candidate and the interviewer. AI use is not an edge case that we tolerate. It is an explicit condition of the interview.

We evaluate how candidates turn incomplete requirements into a precise task. We watch whether they challenge the model’s output, how they reason about trade-offs, and whether they can explain the code that was generated. We also verify that they retain enough independent knowledge to judge the model rather than simply follow it.

The problems are framed as realistic engineering scenarios: optimizing a student dashboard service, implementing the core of a web crawler, or designing a component under concrete performance and reliability constraints.

Each interview has two scored phases: design and requirements gathering, followed by live coding.

During the design phase, we do not score whether the candidate used AI. We score how they used it.

Strong candidates narrow the problem before asking for a solution. They identify missing constraints, question assumptions, and support their choices with rough calculations around throughput, latency, storage, and failure modes.

Weak candidates accept the first answer at face value. They cannot explain why a particular approach was chosen, or they describe an architectural decision as “obvious” without being able to defend it.

Architectural fundamentals are still evaluated directly. The purpose is not to test memorization. We need to know whether the candidate understands the system well enough to recognize when the model is leading them in the wrong direction.

The same principle applies during live coding. Candidates may use AI for any part of the solution, but they remain responsible for the code. The interviewer can select an arbitrary section and ask:

Could this be implemented differently, and why would you choose one approach over another?

The candidate answers without AI.

Every session also includes five to ten minutes of unassisted coding. The candidate writes a moderately complex function using concepts such as timeouts, retries, cancellation, or context handling.

If they cannot work with the fundamentals independently, that phase fails regardless of how polished the AI-assisted solution looks.

For senior and staff-level roles, we add another dimension. We look at whether candidates can turn ambiguous intent into a testable specification, redesign an existing process around AI-assisted execution, define evaluation and failure criteria, and build permissions and accountability into the system from the beginning.

This change does not end when someone is hired.

Routine implementation work has historically helped junior engineers develop judgment. They learned by writing ordinary code, seeing it fail, debugging it, and gradually building a mental library of failure modes.

If agents absorb more of that routine work, we cannot assume that the same judgment will develop automatically. Unassisted exercises, supervised exception handling, rotations, and structured review have to become deliberate parts of the engineering operating model.

AI changes what an engineer can produce. Our interviews and development systems must measure whether the engineer can still understand, challenge, and own the result.

Trust Is Earned One Workflow at a Time

New AI tools appear faster than a normal enterprise adoption cycle can comfortably process them. Evaluation, controlled pilots, and gradual rollout are still necessary, but the technology may change materially before a traditional review process is finished.

The answer is not to skip evaluation. It is to evaluate under real operating conditions without handing over authority too early.

Most AI workflows today are still assisted: an agent does the work, and the person validates the result. When we find a process that could plausibly become more autonomous, we run the AI-assisted version alongside the existing manual path.

The agent handles what it can. The old process covers the gaps and retains final authority.

One example is crash analysis in one of our larger product verticals.

The mobile stability team used to work through top crashes manually. An engineer would read the stack trace, identify the responsible module, find the owner, create a ticket, wait for a fix, and then verify the result in a later release.

Working with the product and stability teams, we built dedicated AI skills for iOS and Android. By a skill, we mean a reusable, parameterized workflow designed for a specific class of work, rather than a general chat prompt.

The workflow can now read crash reports and stack traces, identify the likely responsible module, suggest an owner, prepare a draft pull request, and connect the proposed fix to release validation.

The team still decides whether the diagnosis is correct, whether the change should be accepted, and whether the workflow has earned more authority.

While the AI component is being tuned, the manual process continues to run in parallel. That gives us evidence from real production conditions without making the business process depend on unproven autonomy.

Average accuracy is not enough. The cases that determine whether a workflow is trustworthy are often the unusual ones: a misleading ownership signal, an unfamiliar stack trace, or a fix that works locally but fails during rollout.

We call the principle behind this Tail Reliability. Autonomy increases only after a workflow has demonstrated stable performance across the long tail of edge cases that can hurt us, not merely on the average case.

The reliability bar is different for every workflow. An agent drafting documentation should not be evaluated like an agent initiating a rollback. The potential damage, reversibility, observability, and available human intervention all matter.

We also do not treat autonomy as a permanent achievement. A workflow may move from assisted to supervised operation after it passes evaluation. But a new model, prompt, tool integration, or vendor version does not automatically inherit the previous level of trust. If the workflow regresses against its evaluation suite, its autonomy is reduced until reliability is demonstrated again.

Make Good AI Use Repeatable

Giving every engineer access to an AI tool is not an AI transformation. Access matters, but the larger question is whether good use can spread beyond the people who discovered it first.

We approach enablement through tooling, education, and local role models.

At the company level, we have opened agentic AI tools to all engineers. We want engineers to have access to capable and efficient models. But access to a model is not where an engineering organization creates lasting advantage. The advantage comes from the framework around the model and the domain context connected to it.

We maintain a common repository of reusable AI skills. Teams contribute workflows for code review, test generation, incident analysis, crash investigation, and other recurring engineering tasks. What one team figures out becomes infrastructure for everyone else.

The second part is education.

inDrive has a structured AI education program for engineers. It includes a foundational course, an advanced cohort, and specialized tracks for QA, mobile, and backend development.

The foundational course is part of engineering onboarding. Every new engineer starts with the same basic context: how the tools work, where they tend to fail, how we expect them to be used, and where human judgment remains mandatory.

The goal is not to teach people how to generate more code. It is to develop specification literacy, systems thinking, evidence-based decision-making, and calibrated trust.

Engineers need to know what can be delegated, what must be checked, and when the model is operating outside the context it needs to make a useful recommendation.

The third part is role modeling.

Inside teams, that role is filled by people we call AI champions. They do more than use the tools frequently. They receive early access, break workflows, investigate failures, debug integrations, and explain what they learn to their teammates.

Documentation can explain what a tool is supposed to do. A trusted colleague can show the team how it fails, how to recover, and when not to trust it. That embedded experience is difficult to replace with internal newsletters or one-time training sessions.

Tech Platform and the champion network provide the shared capabilities and trusted context. Product teams apply those capabilities to their own domain problems and remain accountable for the outcome.

This does not turn specialists into generic generalists. We still need deep expertise. But engineers increasingly need enough breadth to understand how a decision moves from an initial request, through an agent and a set of tools, into a production outcome.

Build a Platform Agents Can Actually Use

Consider a developer investigating a failed production rollout.

The information they need may be spread across Grafana, Kubernetes, GitHub, the deployment platform, internal documentation, and the ticketing system. The difficult part is often not running a single query. It is assembling enough context to understand what happened.

An agent can help with that work, but only when it can discover the right tools, receive the right context, and act within clearly defined permissions.

API-first design made systems programmable. Agentic systems add another requirement: the interface must also be discoverable and usable by an agent without bypassing the controls of the underlying system.

We describe this direction as MCP-first.

The Model Context Protocol does not replace APIs. It provides a standardized way for AI applications to discover and invoke tools backed by those APIs. MCP is the connection mechanism. Governance, context, permissions, and accountability still have to be built around it.

In our target architecture, we call this surrounding layer the Knowledge Spine.

It has three main parts.

The Protected Knowledge Zone contains stable, curated context that agents must not redefine: architecture principles, security requirements, product contracts, verified runbooks, and strategic guardrails.

Agents may read this information, but they cannot modify it directly.

The Live Knowledge Zone contains operational context: metrics, deployment state, incident timelines, and knowledge created as people and agents execute work. Updates are controlled, attributable, and auditable.

The Orchestrator sits above both zones. It handles identity, routing, access control, tool selection, context injection, and the audit trail for each interaction.

This separation is deliberate. An agent should not be able to rewrite the rules governing its own behavior while it is executing a task.

Our internal Dev Platform, the CI/CD and deployment system used across product teams, has its own MCP server. Through it, an authorized agent can search for services, inspect deployment state, investigate rollout failures, retrieve pipeline context, and initiate a rollback within governed permissions.

Our internal Zero-Code Platform follows a similar pattern. Its MCP layer proxies the existing GraphQL API and authentication flow, allowing product managers to manage widget configuration through an agent without working directly in the administration interface.

The goal is not to replace every UI and CLI with a chat window. The goal is to reduce the cost of assembling context across fragmented systems while preserving the permissions, controls, and auditability of those systems.

An agent can collect evidence and build an operational picture. The underlying platforms still decide what data is available, which actions are permitted, and who is responsible for the result. MCP is how tools connect to the agent. The Knowledge Spine determines what the agent knows, what it may do, and who remains accountable.

Treat AI Capacity Like Infrastructure

Once agents can act across internal systems, AI stops looking like a collection of individual subscriptions. It starts looking like a familiar category: shared infrastructure.

It has to be provisioned, observed, metered, secured, evaluated, and supported.

The cloud analogy is useful here, but it is often reduced to cost.

For a stable and predictable workload, dedicated infrastructure may be cheaper than cloud infrastructure. The same can be true for AI-assisted work. AI is not automatically the cheapest way to perform every task, and token cost should not be confused with the full cost of an engineering outcome.

The more interesting property is elasticity.

A team may need additional capacity for a migration, release-hardening effort, incident backlog, documentation push, or short product experiment. AI capacity can absorb part of that temporary peak without forcing the organization to treat a short-term spike as permanent demand.

Tech Platform’s role begins to resemble the role infrastructure teams played during the original cloud migration. We provide a reliable, observable, governed, and metered layer of AI capacity that product teams can apply to their problems without rebuilding the same foundation independently.

The capacity created by agents does not simply disappear from the organization.

It moves into work that was previously too expensive to prioritize. It also creates new permanent work: evaluation, skill development, knowledge curation, security, observability, and exception handling.

And it should move engineers toward the parts of the job where their judgment matters most.

That is what “augment, not replace” means operationally. The team’s impact grows, but quality is not traded for automation. People remain capable of judging the work, owning the result, and taking over when the workflow reaches an exception it cannot safely handle.

The Baseline Will Keep Moving

The GPT-3.5 interview experiment we ran in January 2023 was small and informal. But it made a larger change visible earlier than we expected.

If a general-purpose language model can reproduce what your hiring process treats as a senior-level signal, that signal is no longer sufficient on its own. The same logic applies to the engineering platform.

When AI changes what individuals can do, the organization has to redesign the environment around their work: the interviews, training, workflows, tools, context, evaluation, permissions, and guardrails. The hard part is not adopting a model. It is moving quickly without outsourcing judgment.

Trust has to be earned by each workflow. Access has to be governed. Context has to be maintained. Engineers have to understand the systems they operate, challenge the output they receive, and remain accountable for what reaches production.

AI transformation is not a project with a completion date. It is a continuous redefinition of the engineering baseline.

The job of an engineering platform is not to help teams catch up after that baseline moves. It is to make the new baseline safe, usable, and repeatable.

The models may be external. The engineering system around them is not.

What Happens to Your Engineering Platform When AI Raises the Baseline was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, I’m Alexey, an iOS engineer at inDrive. So, let’s start with a simple question: why build a CLI if the admin panel already works? The answer is simple — some tasks are faster in the terminal, and AI agents need a command-based interface.

Why build a CLI on top of a working admin panel

We have BDU — Backend-Driven UI: widgets, datasets, frames, translations. There’s an admin panel where all of it is edited through a Figma-like interface. The panel works, and our design-engineering team lives in it every day. But a few pain points kept recurring.

The first: routine edits. Change a background color, move a heading, fix a translation key — in a form that’s five to ten clicks: open the form, scroll, find the field, save. From the keyboard, via pull → edit the JSON in your IDE → push, the same edits take seconds.

The second: AI agents. When I sit down in Claude Code and ask it to “take this reference widget and build a similar one for a new design,” the agent can’t click through forms. It needs commands with stdin/stdout and a predictable contract.

The web admin panel was never built for these scenarios in the first place. It stays right where it is — for onboarding, for visual debugging, for designers. The CLI complements it; it doesn’t replace it.

The thesis: the CLI is the third interface

Historically an internal service had two “faces”: an HTTP API for other services and a web admin panel for people. The CLI was usually a thin wrapper over the API — “for developers too lazy to open a browser.”

Since 2025 the picture is different. The CLI is an interface that fits three audiences equally well: a developer in the terminal, a script in CI, and an AI agent over stdin/stdout. All three share one contract: commands, flags, exit codes. A web portal has to either expose a full API or lose on composability. The CLI composes by default: pipe, jq, xargs, an exit code folded into a condition.

I’m not discovering anything with this framing — over the past year every other tech blog has written about the “terminal renaissance.” The New Stack covered the agentic CLI era, and Dev.to covered Terminal Renaissance 2026, framing the CLI as the only interface that supports the full spectrum: manual, automated, and AI-augmented. I simply agree, and I want to show what it looks like from inside a small team.

This matters especially for small teams. We’re 15 people, but only one backend engineer (two, as of recently) for all of BDU — any feature that hits the API goes through them. “Build your own CLI” used to be a quarter-long project: flag parsing, help text, completions, distribution, documentation. Now it’s a single evening to a working MVP, and that changes the math: you can fit it in without pulling backend off their work.

Why building a CLI is dramatically easier now

“Your own CLI” used to require a separate skill set: parsing argv, generating help text, completions, a distribution matrix. Today almost all of that is handled by libraries and AI codegen.

Mature frameworks. Apple’s swift-argument-parser, cobra in Go, clap in Rust, Click/Typer in Python, oclif/commander in TypeScript. They all give you subcommands, typed flags, help-text generation, and shell completions for free. You don’t write an argv parser by hand.

Ready-made TUI libraries. Spinners, progress bars, dialogs in a consistent style — with no hand-written ANSI escape codes. Noora from Tuist in Swift, Bubble Tea from Charm in Go, Ratatui in Rust, Ink in TypeScript. Five years ago terminal UI was crafted by hand; now it’s two imports and a loop.

AI codegen eats boilerplate. Subcommand stubs, flag parsing, test tables, — help text, input validation — this is textbook CLI boilerplate, and it’s exactly where an LLM is strongest. The engineer thinks about architecture and UX, not about what to name the fourth flag.

Distribution collapsed to a single binary. A static binary via mise, brew tap, npx/uvx, or a curl installer. No deb/rpm/msi matrix across five operating systems, no runtime to ship. For an internal tool: curl … | sh behind the internal network.

When I built ExFig earlier — a fork of figma-export that ships Figma assets into code — I walked this same path (there’s an article on it, link to be added after it ships). Each successive CLI came together faster: ExFig took a weekend to v1; zc took one evening.

Case study: zero-code — from an evening MVP to v1.4

zero-code (zc) is our CLI for working with BDU. Stack: Swift 6.2 + swift-argument-parser + Noora + Sparkle for auto-updating the menu-bar app. It’s the same stack I’d used for ExFig — zero learning curve, reusing what I already knew.

The first version came together in one evening in Claude Code. It had zc auth login, zc dataset pull <uuid>, zc dataset push, and zc dataset diff. That was enough for the team to stop opening the admin panel for routine edits. Pull the JSON, open it in the IDE, edit, push, QR to the phone, check on-device — the whole loop takes seconds.

Over ten weeks, by v1.4, the CLI had grown to ~44 commands across 13 noun groups. BDU entities — dataset, frame, group, provider, sender, translation, image, illustration, vertical; utility ones — auth, stat, update, completion. Verbs repeat across entities — pull, push, diff, validate, clone, archive, list, create, deeplink. Batch operations appeared (pull-all, push-all), along with QR/deeplink generation for every entity and multi-environment support with file-based authentication.

A few of the less obvious pieces. Dual-format output: pretty JSON when stdout is a TTY (a human is reading), and TOON when stdout goes to a pipe or a redirect (an agent or a script is reading). TOON saves 30–60% of tokens on long lists — which matters when it’s an LLM agent calling that same pull. Shell completions for zsh/bash/fish — swift-argument-parser emits them with a single command. And Sparkle updates for the ZC.app menu-bar app, so the team doesn’t chase down new versions by hand.

Feedback: a widget in an evening

In April, a colleague from a neighboring team picked up the task of building a new shelf widget for Tambo from a Figma design. Here’s a shortened version of his Slack comment:

I gave the agent a prompt: “build a widget in prod for this Figma design, use ryadom_shelves_categories_* as a reference, and here are the deeplinks for the categories.” From the reference it assembled a shelf with the right copy, created the keys in Crowdin, and I asked it to change the text type — it got it on the second try. I uploaded the images myself and set the events and payloads from the reference.

The widget from ticket SI-484: a Figma design → a prompt with a reference → a finished shelf in prod, with only translation keys and images left for the engineer to handle.

What the CLI won’t replace

The CLI beats the admin panel on repetitive tasks, on scriptable scenarios, and when working with an agent. But there are three places where the panel stays mandatory.

• Discovery. A new person opens the admin panel and sees which entities even exist. The CLI doesn’t give you that — until you know what a frame or a provider is, zc — help won’t help.
• Visual debugging. Previewing a widget across platforms, checking dark theme, checking RTL — that’s done with your eyes, not through a JSON diff.
• Non-engineers. Designers and managers are one of the core audiences of the web panel. A terminal and JSON are an extra layer of tooling for them; there’s no point switching their familiar interface for a handful of edits a month.

So the CLI and the admin panel live in parallel rather than crowding each other out. The panel covers onboarding, visual debugging, and non-engineers; the CLI covers repetitive edits, scriptable scenarios, and the AI agent. Both interfaces talk to the same backend, so the choice is driven by the task, not by the person.

One more thing: zc is a local CLI. It’s not an MCP server — agents call it like any ordinary program, through bash. A separate remote MCP for the admin panel came later, as the next step in the story, but that’s a different article.

Takeaway

“Build your own CLI” in 2026 isn’t a quarter-long project. It’s an evening for the MVP and a week or two of polish, if you pick the right frameworks. For teams without a dedicated DX engineer, that means an internal first-party tool can be built in one person’s spare time, instead of being slotted into next half-year’s roadmap.

And it cuts both ways: the CLI speeds up the engineer, speeds up CI, and opens the service to AI agents — with no separate API layer and no separate integration. When I wrote zc, I wasn’t planning “let’s build an agent interface.” I was building a decent CLI; the agent interface came for free. The CLI has stopped being a thin wrapper over the API — it’s now a proper third face of an internal service. If your service doesn’t have one yet, an evening to experiment is cheaper now than it’s ever been.

The CLI Is the Third Interface: Ship a Team Tool in One Evening was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Real SLA numbers, the legal quirks of hiring in Brazil, and a follow-the-sun rotation that actually works for 24/7 incident support.

TL;DR

• Problem: A 3-person incident support team on a 5/2 schedule in Kazakhstan had a 72-hour SLA — anything filed after hours or on weekends waited until Monday.
• Solution: A three-country follow-the-sun rotation (Kazakhstan → Brazil → Malaysia), built without any night shifts.
• Result: SLA dropped to 2 hours for normal-priority tickets and 1 hour for critical incidents, with full 24/7 coverage.
• Bonus: What Brazilian labor law (CLT) requires you to know before you build a shift schedule there.

The Problem: A 72-Hour SLA and No Night Coverage

Friday, 18:00. The workday ends, the team heads home — and that’s exactly when the app breaks. Users hit a “Failed to load” error when accepting an order. First-line support runs 24/7 and keeps receiving tickets, but the technical escalation team works a standard 5/2 schedule. The result: unresolved tickets pile up until Monday morning.

At the time our second-line technical support team was formed, there was no formal SLA. In practice, ticket processing time had crept up to 72 working hours — and we couldn’t even say how many tickets were coming in, because there was no ticketing system to track volume.

Starting point:

• 3 engineers based in Kazakhstan (Almaty, UTC+5)
• Schedule: 5/2, 9:00–18:00
• Coverage split by vertical: Intercity/couriers/freight, US/Australia, and Ride-Hailing
• No dashboards, no volume tracking — a fully reactive setup
• Anything filed evenings or weekends sat until Monday

Why We Ruled Out Night Shifts

As inDrive expanded into the US, Australia, and the broader LATAM region, ticket volume grew — and so did the temptation to just add night shifts in Kazakhstan. We deliberately avoided that path, for three reasons:

1. Burnout. Night shifts wreck sleep and long-term productivity. Daytime work keeps the team healthier and more sustainable.
2. Incident quality. A daytime brain reads logs and admin panels far more accurately than a sleep-deprived one — especially on complex incidents.
3. Cross-team collaboration. Working daytime hours means overlapping with HQ teams, which builds context and communication skills that a graveyard shift never gets.

Instead, we built a follow-the-sun support model: hire in time zones that naturally cover the gaps, so every engineer works their own local daytime.

Iteration 1: Kazakhstan, 2/2 Shifts

Goal: Extend coverage beyond the standard 5/2 day without burning out the existing team.

What we did:

• Hired 2 additional specialists in Kazakhstan on a 2/2 rotation
• Shift window: 03:00–15:00 UTC (08:00–20:00 Almaty time, UTC+5)
• Routed the full first-line escalation stream to this team

Why Kazakhstan again: Most of the ticket volume in this hemisphere falls inside Kazakhstan working hours, and we already had strong first-line support talent there to promote internally.

Result: SLA dropped from 72 hours to roughly 16 hours. Evenings in Almaty were now covered — but early morning and overnight for Asia, plus mornings for other regions, were still blind spots.

Iteration 2: Brazil, 1/1 Shifts

Goal: Cover the overnight gap in Almaty and cut response times for LATAM and adjacent regions.

What we did:

• Hired specialists based in Brazil
• Shift window: 11:00–23:00 UTC (08:00–20:00 Recife time, UTC−3)

Result: Night and early-morning coverage improved significantly. One gap remained: a roughly 4-hour seam between the end of the Brazil shift and the start of the Kazakhstan shift (04:00–08:00 Almaty time).

Iteration 3: Malaysia — Closing the Loop

Goal: Close the remaining early-morning gap and strengthen coverage for the Asian region.

What we did:

• Hired specialists in Malaysia
• Shift window: 00:00–12:00 UTC (08:00–20:00 Kuala Lumpur time, UTC+8)

Result: Full, continuous 24/7 coverage across three countries — with zero night shifts anywhere in the rotation.

Final Numbers

P.S. — What to Know Before You Hire in Brazil for Shift Work

Brazilian labor law (CLT — Consolidação das Leis do Trabalho) has two provisions that directly affect shift scheduling and staffing math:

1. The 12×36 schedule is the legal norm. Under Article 59-A of the CLT (upheld by the Brazilian Supreme Court, STF), employees can legally work 12 consecutive hours followed by 36 hours of rest. In practice: no single person can staff every day — you need paired slots by design.
2. Mandatory Sunday rest. Under Law №605/49, employees must get a rest day that falls on a Sunday at a defined frequency — once every 7 weeks for men, once every 15 days for women (per Article 386 of the CLT). This typically forces a full Saturday+Sunday weekend roughly once every two months, which has to be built into the rotation in advance.

Takeaway for planning: budget a staffing buffer from day one. If you hire “just enough” people to cover the hours on paper, the first mandatory rest period will blow a hole in your schedule.

If you’re building or scaling an incident support rotation, the short version is: don’t add night shifts — add time zones. What gaps has your team hit trying to cover 24/7?

Follow-the-Sun Support: How inDrive Cut Incident SLA from 72 Hours to 2 Hours — Without a Single… was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi! I’m Kirill Presniakov, and together with Vladimir Pchelyakov we’re iOS engineers at inDrive. Mobile performance rarely breaks all at once. It degrades quietly — one release, one screen, one percentile at a time. By the time users start complaining, the regression has often already become part of the product experience.

The right place to catch regressions is before they reach production. We do run automated performance tests with XCUITest — you can read about that approach here — but no pre-release suite covers everything. Real devices, real traffic, and real network conditions always surface things that synthetic tests miss. That means you still need to know what happened after every release ships.

We use two main data sources for iOS performance. The first is our own telemetry — performance events collected from mobile clients and stored in Elasticsearch. This covers a range of signals including screen load times, Time to Interactive, and other key interaction metrics across different user roles, markets, and order types. You can read more about our approach to mobile performance instrumentation here.

The second is native iOS metrics from Apple, available in Xcode Organizer. These include:

• launch time
• hang rate
• peak memory usage
• disk writes
• CPU activity
• battery usage
• app size

All of this data can be explored in Xcode Organizer — but only manually: open it, look through the numbers, close it. No history across releases, no alerts, no automation. In this article, we focus on the second source — how we automated it end to end, set up automatic alerts, and eliminated the need to manually check Xcode Organizer altogether.

With Xcode 27 (announced at WWDC26), Apple added a new Storage category to Xcode Organizer — tracking App Size and Documents & Data footprint over time. The metric also started appearing in the perfPowerMetrics API response. Our pipeline had been running for several months by that point, and when Apple rolled out the new metric, it just showed up and worked - no code changes, no pipeline updates needed.

The end result of the workflow described below is a fully working pipeline plus a set of dashboards built on top of the data it produces — giving us historical trends, per-release comparisons, and a single place to monitor iOS performance across all our apps.

How It Works: Pipeline Overview

Before diving into the implementation details, here’s the big picture. The system is built around two n8n workflows that run independently every day.

The Collection Workflow talks to the Apple App Store Connect API. It authenticates, fetches the list of recent app versions, then loops over each one — pulling raw performance metrics, computing a statistical anomaly score (z-score), and writing the result to Elasticsearch (ES for short — we’ll use that abbreviation throughout the article).

The Alerting Workflow runs separately. It reads the metrics already stored in ES, computes a color-coded Stability Indicator per metric, filters out anything that was already reported, and delivers new signals to Slack and Jira.

Getting Data From App Store Connect API

Working with Apple’s App Store Connect API starts with authentication.

Access to the App Store Connect API requires a Developer role or higher in the Apple Developer Program. Only a user with the Admin role can create an API key — it’s issued in App Store Connect and works for the entire team. If you already have the necessary permissions, let’s move on to authentication.

The Apple API requires a JWT token signed with a private key using the ES256 algorithm (ECDSA + P-256). Detailed instructions for creating keys and generating tokens are in the official documentation: Creating API Keys and Generating Tokens for API Requests.

You need three things to generate the token:

• keyID — the key identifier, found in App Store Connect → Users and Access → Keys
• issuerID — your organization’s Issuer ID, a UUID string like c055ca8c-e5a8–4836-b61d-aa5794eeb3f4, on the same Keys page. This is not the Team ID — they’re often confused
• .p8 file — the private key in PEM format, downloaded once when the key is created

The JWT payload structure looks like this:

{
  "iss": "<issuerID>",
  "iat": 1716300000,
  "exp": 1716301200,
  "aud": "appstoreconnect-v1"
}

The token lives for a maximum of 20 minutes. We generate it in a separate n8n workflow and pass it as an input parameter to the main one — this isolates and reuses the authentication logic.

One non-obvious detail when implementing this in Node.js is the dsaEncoding parameter when signing:

const signatureDER = sign.sign({
  key: privateKeyPEM,
  dsaEncoding: 'ieee-p1363',
});

By default, Node.js returns the signature in DER format (ASN.1), but Apple expects IEEE P-1363. Without this parameter, the token looks valid but Apple returns 401 — and tracking down the cause without knowledge of ES256 internals is far from obvious.

Fetching Native iOS Performance Metrics

The endpoint for native metrics is perfPowerMetrics. The full API reference is available in the official Apple documentation. Getting to the endpoint requires a few steps:

1. Request the list of app versions:

GET /v1/apps/{appId}/appStoreVersions?filter[platform]=IOS&fields[appStoreVersions]=versionString,appStoreState&limit=10

The response contains a data array with versions. From each element we need versionString, appStoreState, and links.self — the last one is used to build the URL for the next request:

{
  "data": [
    {
      "id": "abc123def456",
      "attributes": {
        "versionString": "5.169.0",
        "appStoreState": "READY_FOR_SALE"
      },
      "links": {
        "self": "https://api.appstoreconnect.apple.com/v1/appStoreVersions/abc123def456"
      }
    }
  ]
}

2. For each version, fetch the associated build — the URL is constructed as links.self + /build:

GET /v1/appStoreVersions/{versionId}/build

The response contains many fields, but we only need two: data.id (the build identifier) and the perfPowerMetrics link in relationships:

{
  "data": {
    "id": "1b3aa84f-af15-4f37-a92b-dd7dbae8dacf",
    "attributes": {
      "version": "2604080504"
    },
    "relationships": {
      "perfPowerMetrics": {
        "links": {
          "related": "https://api.appstoreconnect.apple.com/v1/builds/1b3aa84f-af15-4f37-a92b-dd7dbae8dacf/perfPowerMetrics"
        }
      }
    }
  }
}

3. Extract the perfPowerMetrics link and fetch it:

GET https://api.appstoreconnect.apple.com/v1/builds/{buildId}/perfPowerMetrics

The response is a JSON with a productData → metricCategories → metrics → datasets → points hierarchy. Apple provides only two percentiles — p50 and p90. Each data point is a metric value for a specific version and device:

{
  "productData": [
    {
      "metricCategories": [
        {
          "identifier": "LAUNCH",
          "metrics": [
            {
              "identifier": "launchTime",
              "unit": {
                "displayName": "ms"
              },
              "datasets": [
                {
                  "filterCriteria": {
                    "device": "all_iphones",
                    "percentile": "percentile.ninety"
                  },
                  "points": [
                    {
                      "version": "5.167.0",
                      "value": 462.1
                    },
                    {
                      "version": "5.168.0",
                      "value": 471.8
                    },
                    {
                      "version": "5.169.0",
                      "value": 487.2
                    }
                  ]
                },
                {
                  "filterCriteria": {
                    "device": "all_iphones",
                    "percentile": "percentile.fifty"
                  },
                  "points": [
                    {
                      "version": "5.169.0",
                      "value": 312.5
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "insights": {
    "regressions": [
      {
        "metricCategory": "LAUNCH",
        "metric": "launchTime",
        "latestVersion": "5.169.0",
        "maxLatestVersionValue": 487.2,
        "referenceVersions": ["5.167.0", "5.168.0"],
        "summaryString": "Launch time increased 8% in version 5.169.0"
      }
    ]
  }
}

After this request, n8n applies two checks. The first is for the presence of metrics: for a fresh version, productData may be empty — Apple simply hasn’t accumulated enough data yet. The second is for an HTTP error in the response: this means the version was revoked and the resource is unavailable. In both cases, the version is skipped and the workflow moves to the next one. More on this in the pitfalls section.

We filter only device = all_iphones. The last 4 versions are a sufficient horizon for trend calculation.

Tracking the Current Version Between Runs

Before diving into the workflow logic, it’s worth explaining what z-score is and why we need it.

Z-score is not the standard deviation itself — it’s a measure of how far the current value is from it. The formula: z = (x − μ) / σ, where x is the current p90 value, μ is the mean of historical data, σ is the standard deviation. Z-score shows how many “sigmas” the current result deviates from the norm: z = 0 means “exactly on the mean”, z = 2 means “two standard deviations worse than normal.” This coefficient is what we store in the ES document and use for the semaphore metric assessment. More details in the Z-score section below.

We only compute z-score for the latest current version, for two reasons. First, z-score is not a static value: it’s recalculated each time new historical data accumulates. Computing it for all versions on every run would leave ES full of documents with stale and contradictory values. Second, we only care about the current state of the product — alerting on a version that has already been superseded by two newer ones has little value.

The workflow runs daily and fetches the last 10 versions each time. The Track Latest Version node remembers the last processed version and compares it to the current one — so z-score is only computed where needed.

We do this via $getWorkflowStaticData(‘global’) — n8n’s built-in mechanism for persisting data between workflow runs:

function compareVersions(a, b) {
  const pa = a.split('.').map(Number);
  const pb = b.split('.').map(Number);

  for (let i = 0; i < Math.max(pa.length, pb.length); i++) {
    const diff = (pa[i] || 0) - (pb[i] || 0);
    if (diff !== 0) return diff;
  }

  return 0;
}

const staticData = $getWorkflowStaticData('global');
const currentVersion = $('Split Out').item.json.attributes.versionString;
const storedVersion = staticData.lastActualVersionWithMetrics ?? null;

let shouldComputeZScore = false;

if (!storedVersion) {
  // First run — save the version and compute z-score
  staticData.lastActualVersionWithMetrics = currentVersion;
  shouldComputeZScore = true;
} else {
  const cmp = compareVersions(currentVersion, storedVersion);

  if (cmp > 0) {
    // New version released — update stored version
    staticData.lastActualVersionWithMetrics = currentVersion;
    shouldComputeZScore = true;
  } else if (cmp === 0) {
    // Same current version — compute z-score again
    shouldComputeZScore = true;
  }

  // cmp < 0 — version is older than stored, z-score is not needed
}

staticData is mutated in memory and n8n persists it between runs. For older versions, _shouldComputeZScore = false — they skip the ES history query and z-score calculation and are written to the index without that field.

Handling Versions Without Metrics

Not every version in the App Store has performance data. Two scenarios are possible:

Data hasn’t accumulated yet. Apple starts aggregating metrics not immediately after a release, but as users install and run the app. For a fresh version, perfPowerMetrics will return a successful response but with an empty array: productData: [].

The version was revoked. If a release was rolled back, the resource may become unavailable — and Apple will return an HTTP error.

We handle both cases. The HTTP error is caught by the onError: continueErrorOutput parameter on the GET perfPowerMetrics node — instead of crashing, the error is routed to a separate output and control returns to the version loop. The empty array is handled by the next Has metrics? node, which checks productData.length > 0 and also skips the version. In both cases, the workflow continues processing the remaining versions normally.

Apple Insights: Built-in Regression Analysis

The perfPowerMetrics response contains not only raw metric values, but also an insights block with two arrays: regressions — metrics where Apple detected degradation — and trendingUp — metrics with a sustained upward trend. Each entry contains a human-readable summary and per-percentile breakdown:

{
  "insights": {
    "regressions": [
      {
        "metricCategory": "LAUNCH",
        "metric": "launchTime",
        "latestVersion": "5.166.0",
        "referenceVersions": [
          "5.162.0",
          "5.163.0",
          "5.164.0",
          "5.165.0"
        ],
        "summaryString": "Top (90th percentile) and typical (50th percentile) Launch Time trended up between app versions 5.162.0 to 5.166.0 on iPhone (All). The maximum value for the latest app version is at 5871.80 ms and increased by 21% compared to the average for previous 4 app versions.",
        "populations": [
          {
            "percentile": "percentile.fifty",
            "device": "all_iphones",
            "referenceAverageValue": 1477.98,
            "latestVersionValue": 1774.2,
            "deltaPercentage": 20
          },
          {
            "percentile": "percentile.ninety",
            "device": "all_iphones",
            "referenceAverageValue": 4856.65,
            "latestVersionValue": 5871.8,
            "deltaPercentage": 21
          }
        ]
      }
    ],
    "trendingUp": [
      {
        "metricCategory": "BATTERY",
        "metric": "onScreen",
        "latestVersion": "5.166.0",
        "summaryString": "Top (90th percentile) ScreenOn Location Drain trended up between app versions 5.162.0 to 5.166.0 on iPhone (All)."
      }
    ]
  }
}

insights.regressions is the easiest way to get started with performance analysis: no infrastructure needed — Apple already compared versions and wrote the summary. If you’re just starting to build a similar system, this is a good entry point.

However, there are fundamental limitations. Apple only covers its own 6 native categories — it can’t see our custom TTI metrics from Elasticsearch. The detection algorithm is a black box: Apple decides what counts as a regression and what doesn’t. We don’t know the thresholds and can’t tune the sensitivity for our product’s specifics — for some metrics a 5% increase is critical, for others even 20% is within normal range.

In essence, we’re building the same mechanism manually, but with full control: we compute the z-score ourselves, set our own thresholds, and decide what counts as critical versus normal noise. And it works not just for Apple’s metrics, but for all our telemetry in one pipeline.

System Architecture

Having understood the data sources, we move to the next step — how to assemble everything into a unified system. We don’t just need to collect metrics; we need to store them, detect anomalies, and deliver signals to the team. That’s what the architecture below describes.

The system consists of two layers:

Data collection layer — a Go service for TTI metrics from our telemetry, and an n8n workflow for metrics from the Apple API.

Alerting layer — an n8n workflow that queries Elasticsearch daily, computes metric statuses, sends notifications to Slack, and creates Jira tasks.

All data is stored in a single ES index clairvoyance-metrics-traces. This approach allows building unified dashboards in Kibana and using the same queries in both the Go service and n8n. For each metric we’ve set up dedicated dashboards — with version-over-version dynamics, p50/p90, and a goal line where Apple provides one. They’re used for in-depth trend analysis when alerting has already flagged an issue or the team wants to see the big picture.

Apple API → n8n (collection) → Elasticsearch ← Go service (TTI)
                                             ↓
                                    n8n (alerting) → Slack, Jira

Data Collection and Storage in Elasticsearch

The ES Document

Each aggregated result is stored as a separate ES document. The structure is unified for both sources — Apple API and our TTI telemetry. This is a deliberate architectural decision: one index for all metrics means unified Kibana dashboards, identical queries in the alerting workflow, and the ability to compare Apple metrics and TTI in one place.

{
  "@timestamp": "2024-05-20T12:00:00.000Z",
  "name": "AggregationAppStorePerfomanceMetrics",
  "metric": "LAUNCH_launchTime",
  "platform": "iOS",
  "aggregation": {
    "interval": 24,
    "release_number": "5.168.0",
    "timestamp": 1716163200000,
    "build_id": "abc123",
    "build_version": "5.168.0"
  },
  "payload": {
    "p50": 312.5,
    "p90": 487.2,
    "goal": "fair",
    "p90_z_score": 1.84
  }
}

The name field is the document type. It’s used to filter queries: AggregationAppStorePerfomanceMetrics — data from the Apple API, AggregatedTTI — our telemetry. The alerting workflow doesn’t know and doesn’t need to know where the data came from — it works with a unified structure.

The goal field in payload is a target value that Apple returns only for the Animation metric. Apple defines a threshold benchmark (poor/fair) for it and passes it directly in the response. We simply store this value as-is to display it as a goal line in Kibana.

p90_z_score is stored in the document at write time — this is a deliberate choice. The value is fixed against historical data at a specific moment and never changes. Meanwhile, the semaphore indicator (🔴🟡⚪) is computed at query time via runtime_mappings. This split means: if tomorrow we decide to change the Critical threshold from 3.0 to 2.5 — we change one line in the workflow, no need to rewrite thousands of ES documents.

Z-score as an Anomaly Measure

Z-score shows how much the current value deviates from the historical norm in units of standard deviation. Formula:

z = (x - mean) / stddev

We compute it on p90, not p50. Tail values better reflect the real experience of struggling users: if p90 increased, one in ten users feels it right now, even if the median hasn’t changed.

Before computing z-score, the Get History node fetches the last 10 documents for the same metric and platform from ES — that’s the historical baseline. Each element in hits is an ES document in the standard response format, where _source contains the original document and payload.p90 is the stored percentile value.

const historicalP90 = hits
  .map(h => h._source?.payload?.p90)
  .filter(v => v != null && typeof v === 'number');

function mean(arr) {
  return arr.reduce((sum, v) => sum + v, 0) / arr.length;
}

function stddev(arr) {
  const m = mean(arr);
  const variance = arr.reduce(
    (sum, v) => sum + (v - m) ** 2,
    0
  ) / (arr.length - 1);

  return Math.sqrt(variance);
}

const meanVal = mean(historicalP90);
const stddevVal = stddev(historicalP90);

const zScore = historicalP90.length >= 2 && stddevVal > 0
  ? (latestP90 - meanVal) / stddevVal
  : 0;

To get a feel for the scale — a concrete example: historical p90 mean = 400ms, standard deviation = 50ms, current p90 = 600ms. Then z = (600 − 400) / 50 = 4.0 → 🔴 Critical degradation. If current p90 = 430ms → z = 0.6 → ⚪ Normal, everything within bounds.

If there are fewer than two historical data points, z-score is forced to 0. The reason is simple: the standard deviation of a single point is mathematically undefined, and with only one value any deviation would look like an anomaly. Better to miss an alert than to generate a false one.

Semaphore Alerting in Slack

Data in Elasticsearch exists — and the team periodically opens Kibana to check trends version over version. But relying solely on manual analysis isn’t enough: between checks, degradation can accumulate unnoticed. We need a mechanism that detects deviations on its own and delivers a signal promptly — not when someone decided to open a dashboard, but as soon as something goes wrong.

That’s what the alerting workflow does. It runs once a day, pulls fresh metrics from ES, computes a semaphore indicator per metric using z-score, and sends to Slack only what has changed since the last check. The team gets a concrete signal — which metric, which version, how bad — and can immediately start investigating rather than scanning dashboards for the problem.

Runtime Mappings in the ES Query

Instead of storing the text indicator in ES (and recalculating it when thresholds change), we compute it at runtime directly inside the search query. ES supports runtime_mappings — Painless scripts that add virtual fields to each document:

{
  "runtime_mappings": {
    "Stability Indicator": {
      "type": "keyword",
      "script": {
        "source": "def z = null; if (doc.containsKey('payload.p90_z_score') && doc['payload.p90_z_score'].size() > 0) { z = doc['payload.p90_z_score'].value; } if (z != null) { if (z >= 3.0) { emit('🔴 Critical degradation'); } else if (z >= 2.0) { emit('🟠 Major degradation'); } else if (z >= 1.0) { emit('🟡 Minor degradation'); } else if (z > -1.0) { emit('⚪️ Normal'); } else if (z > -2.0) { emit('🟢 Minor boost'); } else if (z > -3.0) { emit('🔵 Major boost'); } else { emit('🟣 Exceptional boost'); } }"
      }
    }
  }
}

The second mapping — Stability indicator metric — translates technical metric identifiers into readable names: LAUNCH_launchTime → Launch Time, MEMORY_peakMemory → Peak Memory, BATTERY_batteryUsage → Battery Usage, and so on for all supported metrics.

Both fields are used together when composing the Slack message: the team sees iOS · Launch Time · 5.168.0 · 🟡 Minor degradation instead of the technical identifier LAUNCH_launchTime.

The advantage of this approach: thresholds are stored in one place — in the workflow code. If we decide to change the Critical threshold from 3.0 to 2.5, we change one line rather than rewriting thousands of ES documents.

Alert Deduplication

The most insidious problem in any monitoring system is repeated alerts. If Launch Time for version 5.168 degraded yesterday and we sent a Slack message, nothing has changed today — there’s no point sending it again.

We solve this using DataTable in n8n — a built-in key-value database that survives workflow restarts and persists state between runs. At the start of each workflow run, we load all previously sent alerts:

const existingAlerts = new Set();
const dataTableNode = $('Get Used iOS Alerts');

if (dataTableNode && dataTableNode.all()) {
  dataTableNode.all().forEach(item => {
    const row = item.json;

    if (row && row.alert_key) {
      existingAlerts.add(row.alert_key);
    }
  });
}

The alert key includes platform, metric, indicator, and release:

function createAlertKey(platform, metric, indicator, release) {
  const p = (platform || '').trim().toLowerCase();
  const m = (metric || '').trim().toLowerCase();
  const i = (indicator || '').trim().toLowerCase();
  const r = (release || '').trim().toLowerCase();

  return `${p}-${m}-${i}-${r}`;
}

Including the indicator in the key is intentional. If Launch Time in version 5.168 was 🟡 yesterday but became 🔴 today — that’s a different key, meaning a new event, meaning a new alert. The team needs to know the situation has worsened. Without the indicator in the key, a re-alert on escalating degradation would simply be silenced.

Jira Integration with Smart Deduplication

Why Not Just Slack

Slack is great for real-time notifications, but poor as a tracker. Messages get lost, tasks don’t get assigned. For degradations that require real investigation, we automatically create Jira tickets.

Deduplication Key: One Ticket per Metric per Release

Jira deduplication works differently from Slack. We don’t need to create a new ticket every time the indicator changes — one ticket per metric per release. If a ticket for Launch Time in version 5.168.0 already exists, we don’t touch it again, even if the degradation escalated from 🟡 to 🔴.

The key is built from the release and normalized metric name:

function createJiraKey(release, metric) {
  const r = (release || '').trim().toLowerCase();
  const m = (metric || '').trim().toLowerCase().replace(/\s+/g, '_');

  return `jira-ios-${r}-${m}`;
}

// Example:
// jira-ios-5.168.0-launch_time

Before creating a ticket, we check two sets:

• existingAlerts — keys from DataTable (what was created in previous runs)
• seen — keys we’ve already processed in this run (to handle ES duplicates)

if (existingAlerts.has(jiraKey)) continue; // Already created before
if (seen.has(jiraKey)) continue;           // Already processed this run

seen.add(jiraKey);

If both checks pass — we create the task with a description and a link to the Kibana dashboard.

Jira Ticket Creation Flow

ES data → degradation filter → check existingAlerts and seen
    ↓
Create Jira Issue (one task per metric/release)
    ↓
Save jira_key to DataTable

System Architecture

The final architecture has two main layers.

The first layer collects data:

Apple App Store Connect API → n8n workflow → Elasticsearch
Internal TTI telemetry → Go service → Elasticsearch

The second layer handles alerting:

Elasticsearch → n8n alerting workflow → Slack + Jira

All metrics are stored in a shared Elasticsearch index. This allows us to build unified Kibana dashboards and reuse the same alerting model for different metric sources.

The architecture is intentionally simple.

n8n handles orchestration well. Elasticsearch is already part of the observability stack. Jira and Slack are where engineering teams already work.

The value of the system is not in adding another dashboard. The value is in connecting performance signals to the team’s daily workflow.

Lessons Learned

Missing data is not zero

When Apple does not have enough data for a release, the value may be missing. Treating that as zero would produce meaningless Z-scores and false alerts.

Absence of data must be handled as a separate state.

Fresh releases need patience

Performance metrics are not always available immediately after release. Apple needs time to aggregate real-world usage. The workflow should expect this and continue gracefully.

Deduplication rules should match the channel

Slack and Jira have different purposes.

Slack is for events. Jira is for work.

That is why Slack deduplication includes the status, while Jira deduplication does not. A status change deserves a Slack update, but not necessarily a new Jira ticket.

Statistical alerts need enough history

Z-score only works when there is enough historical data. With too few points, the system should avoid pretending to be confident.

Internal storage has limits

n8n DataTable is simple and convenient, but it is not an infinite event store. If alert history grows too much, old entries need to be cleaned up or moved to external storage.

Simple tools are often enough for a first production version, but their limits should be explicit.

What Changed for the Team

Before this system, checking performance after release required manual attention.

Someone had to open dashboards, compare versions, notice unusual values, and decide whether the change mattered. This process worked only when people remembered to do it.

Now the workflow runs automatically every day.

It fetches fresh performance data, calculates statistical deviation, determines the severity, checks whether the alert is new, sends Slack notifications, and creates Jira tickets for regressions.

For engineers, this means less dashboard babysitting.

For the product, it means performance regressions are less likely to silently become part of the user experience.

For the organization, it means mobile performance becomes a process, not a personal habit.

Conclusion

Performance monitoring is not just about collecting metrics.

It is about building a system that helps teams notice meaningful changes, reduce noise, and act before users are affected at scale.

By combining Apple’s native iOS performance data, internal telemetry, Elasticsearch, Z-score anomaly detection, Slack alerts, and Jira automation, we built a practical monitoring pipeline that fits into the way our engineering teams already work.

The core idea is simple:

Do not wait until users tell you the app became slower. Build a system that tells the team first.

How We Automated Xcode Organizer Performance Monitoring was originally published in inDrive.Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.