How we test and tune face matching for identity verification across races and nations

This is the third post in our identity verification (IDV) engineering series. In Unveiling Face Recognition: From Pixels to Identity we walked through how a face-recognition pipeline works, and in Inside Auto Capture we showed how we capture good document and selfie images on the device. In this post we share our findings around model bias: where it enters a face-matching pipeline, how we measure it across races and nations, what we do when we find a real gap, and what we found when we looked.

The one-second decision

Someone opens a bank’s onboarding flow, photographs their ID card, then takes a selfie. About a second later, our IDV service answers one question: is the face on the document the same face in front of the camera?

That answer is a number. Two face images become two vectors, the distance between them is measured, and the distance is compared against a threshold. Everything that shaped those vectors, above all who was in the training data, shapes who gets waved through and who gets told to try again.

Face recognition is not equally accurate for everyone. In 2019, the U.S. National Institute of Standards and Technology (NIST) evaluated 189 algorithms for demographic effects and found that, for many of them, false-match rates differed by a factor of 10 to 100 between demographic groups. A year earlier, the Gender Shades study showed commercial gender classifiers erring on up to 34.7% of darker-skinned women versus under 1% of lighter-skinned men.

Those studies cover general-purpose face recognition. Identity verification adds twists of its own. In this post you’ll see where bias enters a face-matching pipeline, why document-based verification makes it harder, and two things we actually did about it: we built a benchmark that could see bias and changed our embedding model because it found a real gap, and we evaluated our models on a population under-represented in our training data, only to find that the bias we feared was not the one in the data.

Where the pipeline can go wrong

A face-matching system is a four-stage pipeline. Each stage can introduce demographic error, and errors compound.

1. Detect. A detector finds the face and five landmarks: both eyes, the nose tip, and the mouth corners. Detectors trained mostly on well-lit, high-contrast faces lose recall on darker skin in dim lighting and on tiny, low-contrast document portraits. A missed detection is a rejection before any matching has happened.

2. Align. The five landmarks are warped onto canonical positions in a fixed-size crop, typically 112 by 112 pixels. Landmark errors propagate downstream, and eye-landmark errors are systematically larger for some eye shapes.

3. Represent. A convolutional network maps the aligned crop to a unit-length vector, usually 512 numbers, called an embedding. Modern networks are trained with margin-based softmax losses on millions of identities. This is where most of the bias lives, and the mechanism is simple: the network learns to separate the identities it was shown. If most of those identities share a demographic, the embedding space is finely partitioned there and coarsely partitioned elsewhere. Faces from under-represented groups land closer together, so different people from that group look more alike to the model.

4. Match. The two embeddings are compared with cosine distance and the result is checked against one global threshold. One threshold, many impostor distributions. When a group’s impostor distances sit closer to zero, the same threshold accepts more impostors from that group. A single accuracy number hides this completely. You have to report the false match rate (FMR) and false non-match rate (FNMR) per group.

A short history of embedding methods explains why stage 3 dominates. Eigenfaces (1991), the classical approach, represented a face as weights over principal components, optimized for reconstruction rather than discrimination. FaceNet (2015), the first widely adopted deep-learning approach, trained a network with a triplet loss so that same-person embeddings are pulled together and different-person embeddings pushed apart. ArcFace (2018), the loss function behind most of today’s face-recognition networks, replaced triplets with an angular margin on a classification loss, forcing tighter identity clusters on a hypersphere. AdaFace (2022), a refinement of ArcFace, made that margin adapt to image quality, so blurry or low-resolution faces get a gentler push. Each step made embeddings more discriminative. None of them made embeddings fair by itself, because the loss knows nothing about demographics. It only knows the identities it is shown.

Identity verification is a harder version of the problem

Most face-recognition benchmarks compare two photographs of similar quality. Document verification does not. One side is a live selfie from a phone camera. The other is a portrait printed on a document: taken years earlier, heavily compressed, sometimes grayscale and laser-engraved into polycarbonate, sometimes sitting under a hologram, and often only a few hundred pixels wide once cropped.

This is where “nations” enters the picture as an engineering variable, not only a demographic one. A laminated paper card from one country looks nothing like a polycarbonate card from another, and both look nothing like a passport data page. Our original embedding model was chosen for robustness to age gaps between the document portrait and the selfie. The portrait side of the pair varies by issuing country at least as much as faces vary by ancestry, and a matcher that was never evaluated on that variety will surprise you in production.

Case study 1: a benchmark that could see bias

In early 2025 we committed to NIST-style fairness targets for our 1:1 matcher: FMR and FNMR gaps between the best- and worst-performing demographic groups within a fraction of a percentage point. The first problem was not the model. It was that our benchmark could not measure the target. It covered a few hundred identities from one country, produced one accuracy number, and took two and a half hours per run.

Here is what we changed.

We built a demographically labeled evaluation set from public data. We combined VGGFace2, the class-balanced BUPT-CBFace set, and synthetic celebrity-style faces, then labeled every identity into one of six groups with an off-the-shelf demographic classifier. To reduce label noise we ran the classifier on every image of a person and took a majority vote. Automatic labels are imperfect. They are good enough to find gaps between groups, and not good enough to make claims about any individual.

We sampled hard impostors. Impostor pairs (two different people) were drawn from within the same demographic group. Cross-group impostors are easy and inflate results; the NIST differentials are within-group differentials.

We computed embeddings once. Each face is embedded a single time and pairs are compared from cache. The run dropped from two and a half hours to under a minute, which turned “one experiment a day” into “twenty experiments a day”.

We logged every failure as an image pair. This exposed duplicate identities filed under different IDs in the “different person” set. Part of our measured error was benchmark noise, not model error.

With the benchmark fixed, the baseline model showed a real gap. FNMR is the share of genuine pairs (same person) that the matcher wrongly rejected; FMR is the share of impostor pairs (different people) that it wrongly accepted. On roughly 480 identities per group, at a fixed threshold:

Two caveats. At a few hundred identities per group, a difference of about one percentage point is inside the noise. And measuring FMR at NIST’s operating points, one in a thousand or lower, needs orders of magnitude more impostor pairs than this. The table finds a direction, not a verdict.

What did not work. Test-time augmentation (random crops, flips, color jitter, averaged over ten passes) made results slightly worse. Swapping in other public models such as VGG-Face and GhostFaceNet was worse still. Majority voting across several models gave no gain.

What worked. Two newer open models, AdaFace trained on the WebFace12M dataset and ElasticFace, each beat our original model on the diverse set. The best result came from fusing: concatenate the L2-normalized embeddings of three backbones (our original model, AdaFace, and an ArcFace-style iResNet-100) into a single 1,536-dimensional vector, then compare with cosine distance as before. On a balanced set of 200 identities per group:

The gains are small in absolute terms, which is what you expect from models that are already strong. What matters is that the impostor side improved: the fused model accepts far fewer same-group impostors, which is exactly the failure mode the NIST report flags.

The fusion itself is a few lines:

import numpy as np

def l2_normalize(v):
    return v / np.linalg.norm(v)

def fused_embedding(face, backbones):
    """Concatenate L2-normalized embeddings from several backbones."""
    return np.concatenate([l2_normalize(m(face)) for m in backbones])

def cosine_distance(a, b):
    return 1.0 - float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def match_confidence(distance):
    """Map a cosine distance to an angular score in [0, 1]."""
    return 1.0 - np.arccos(np.clip(1.0 - distance, -1.0, 1.0)) / np.pi

Fusion also solved a production problem for free. The first 512 numbers of the fused vector are exactly the old model’s vector, so every embedding already stored for fraud-face galleries and repeat-face search stayed valid. We migrated online: new sessions write the 1,536-dimensional vector, searches against older records read the 512-dimensional prefix, and nothing went down while the galleries were re-encoded in the background.

And the per-group report that gates every model change since is just as short:

from collections import defaultdict

def per_group_rates(pairs, threshold):
    """pairs: iterable of (group, distance, is_same_person).
    Impostor pairs must be sampled *within* the same group."""
    s = defaultdict(lambda: {"fm": 0, "imp": 0, "fnm": 0, "gen": 0})
    for group, dist, same in pairs:
        if same:
            s[group]["gen"] += 1
            s[group]["fnm"] += dist >= threshold   # genuine pair rejected
        else:
            s[group]["imp"] += 1
            s[group]["fm"] += dist < threshold     # impostor accepted
    return {g: {"FMR": v["fm"] / v["imp"], "FNMR": v["fnm"] / v["gen"]}
            for g, v in s.items()}

def fairness_gap(rates, metric):
    values = [r[metric] for r in rates.values()]
    return max(values) - min(values)

Case study 2: the bias we expected was not the one we found

While evaluating our models on a population with different demographic characteristics than our training data, we had one specific worry. Before matching, the selfie passes a set of quality checks, one of which rejects images where the eyes appear closed. Eyes-closed detectors are a textbook demographic failure mode: a model trained mostly on one eye shape fires falsely on another, and every false trigger is a “please retake your selfie” loop for a legitimate user.

We set the targets before touching any model. For presentation-attack detection we adopted the ISO/IEC 30107–3 metrics (attack presentation classification error rate, bona fide presentation classification error rate, and their average), each capped at 5%. For matching we set FNMR at an FMR of 0.1% to at most 1%, in the NIST 1:1 style. And for parity we required no sub-population to trail the global baseline by more than two percentage points on any of those numbers.

Then we measured on two fronts. We built a labeled evaluation set with sub-populations tagged. And, before waiting for it, we audited production: using the issuing country of the verified document as a coarse proxy for population, we computed eyes-closed and liveness rejection rates for users from that population and for everyone else within the same tenants, with confidence intervals, and reviewed samples by hand.

The result surprised us. Users from the population we were worried about did not have an elevated eyes-closed rejection rate. In the largest tenants the opposite held: everyone else was rejected for closed eyes several times more often, with tight confidence intervals and enough volume that this was not noise. Manual review agreed. We shipped no retraining, and the evaluation set and harness became a permanent monitoring asset.

Three lessons came out of that exercise.

• “This model is biased against group X” is a hypothesis. It is a very plausible one, and you should still test it before you spend a quarter fixing it.
• The check was cheap because the machinery from case study 1 already existed. Fairness testing is a fixed cost that pays back every time a new population comes up.
• The problem we did find was not a demographic one. The check fires more often than it should for everyone, which is a general accuracy question with a product-defined target, not a fairness question, and the target was what we were missing. It went to the product backlog as an accuracy item, not to the fairness ladder below.

What we do when a gap is real

Measuring is half the job. When the per-group report shows a real gap, we work through a fixed ladder, cheapest rung first, and re-run the report after each rung on both the affected group and the global baseline. A fix that helps one group and hurts another does not ship.

1. Recalibrate the threshold. Per-group ROC curves often show that one global threshold sits at a bad operating point for one group. Moving it, or giving a quality check its own threshold, is a configuration change that takes a day.
2. Change the model, not the data. Swap in, or fuse with, a backbone trained on a more diverse corpus. This is what closed the gap in case study 1: no training on our side, a few lines of code, and a measurable drop in same-group false matches.
3. Fine-tune on the under-represented population. Add a labeled split for that group to the training data and fine-tune with class-balanced sampling or a focal loss, so the minority group is not drowned out by the majority. This needs consented data, so we source it from a data partner or, for attack coverage in liveness, generate it synthetically from genuine samples.
4. Replace the model. Some checks ship as inference-only artifacts with no training pipeline behind them. If recalibration does not close the gap, the honest fix is a retrainable architecture, and that is a quarter of work, not a sprint.

Every rung ends the same way: the per-group report on the affected population and on the global baseline, side by side. Case study 1 stopped at rung two. Case study 2 never needed a rung at all, which is the point of measuring first.

What we keep doing

The two case studies turned into habits. If you run a face matcher in production, these are the ones worth copying.

• Report FMR and FNMR per group, never just accuracy, and track the gap between the best and worst group as a first-class metric.
• Sample impostors within a group. Cross-group impostors are the easy case and hide the failure NIST measures.
• Use the document’s issuing country as a production proxy label, with confidence intervals. It is coarse, but it needs no new data collection and catches drift early.
• Treat the threshold as a product decision. Look at the ROC curve per group before choosing it, because one threshold means different operating points for different groups.
• Benchmark the document side. Countries differ in document manufacturing more than populations differ in faces. A portrait engraved in grayscale on polycarbonate is a different input than a color photo on paper.
• Gate every model change on the per-group report. A model that improves the average and widens the gap does not ship.

The bigger picture

Fairness in face recognition is not a dataset you buy once. It is a measurement you build, and then keep running as models change and populations grow. The fused-embedding change in case study 1 was a modest model improvement made possible by a large benchmarking improvement. The evaluation in case study 2 found no model problem at all, and was still worth doing, because now we know rather than assume.

As identity verification becomes the front door to banking, telecom, and government services across the world, “works for most people” is not a shipping criterion. The engineering to get past it is not exotic. It is labeled evaluation sets, honest per-group metrics, hard impostors, a fixed ladder of fixes, and the discipline to look before you fix.

References

• Grother, Ngan, Hanaoka. Face Recognition Vendor Test Part 3: Demographic Effects. NISTIR 8280, 2019.
• Buolamwini, Gebru. Gender Shades. FAT* 2018.
• Deng et al. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. CVPR 2019.
• Kim, Jain, Liu. AdaFace: Quality Adaptive Margin for Face Recognition. CVPR 2022.
• Boutros et al. ElasticFace: Elastic Margin Loss for Deep Face Recognition. CVPRW 2022.
• Wang, Deng. Mitigating Bias in Face Recognition Using Skewness-Aware Reinforcement Learning. CVPR 2020 (BUPT balanced datasets).
• ISO/IEC 30107–3, Biometric presentation attack detection, Part 3: Testing and reporting.

Dvir Dadon leads the IDV Vision team at Transmit Security, the computer-vision and machine-learning group behind document and face verification in the Mosaic identity platform. His team owns the models, benchmarks, and fairness testing described in this post.

Not All Faces Are Matched Equally: Engineering Face Recognition Across Races and Nations was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Magic, and What It Costs

ORMs sell a compelling promise: write TypeScript, get a database. Declare your entities, define your relations, call find() with a filter. The ORM handles the rest. You never have to think about SQL.

That magic is real. But it comes with a cost: your entity and relation declarations deliberately say nothing about index shape or join strategy. They shouldn’t: that’s not what they’re for. The problem is that nothing else tells you either. Most of the time it doesn’t matter. Occasionally it matters enormously, and the ORM gives you no signal either way.

The harder class of problems is those that don’t crash, don’t return wrong data, and don’t show up in your logs until there’s enough data behind them for the cost to become visible. A query that was fast with thousands of rows becomes slow enough to page someone with millions of rows — same code, same tests, different outcome.

The Trap

The ORM generates SQL you never wrote and never reviewed, and it fails in ways that neither CI nor lower environments will surface until scale exposes them:

• Queries that drop tenant scope entirely: a bare WHERE id = :id, or an UPDATE/DELETE, on a tenant-scoped table with no tenantId anywhere in it, no join involved at all.
• Queries badly shaped from the start, unreviewed because the ORM made SQL invisible.
• Search endpoints doing full-table scans because no index was ever considered.
• Cascading relation loads that load every row in a relation; trivial with ten records, a full-table read with millions.
• Delete cascades that fire one query per child row: harmless at small scale, a query storm in production.
• Implicit SELECT DISTINCT injected when combining relations with pagination, forcing a full deduplication pass across the entire result set before LIMIT even runs.

The common thread across all of them: no systematic way to catch any of it before it reached production.

The most structural of these, and the most instructive, is a join between tenant-scoped tables that quietly drops tenantId from the ON clause. It's worth walking through in detail, because it explains why indexes stop being usable at all, not just used inefficiently.

The Join That Makes Indexes Unreachable

Let’s take, for example, a simple join between two tenant-scoped tables. It’s the kind the ORM generates automatically when you load a relation.

Our system is multi-tenant. Every row in every table carries a tenantId column: top-level records, child tables, junction tables, every relation. Tenant segregation is a correctness requirement, not a filter. Because every query filters by tenant, every index leads with tenantId: composite indexes (tenantId, …), primary keys (tenantId, id), and FK relationships structured as (tenantId, userId).

The reason this matters structurally: composite indexes are only usable when the query’s filter covers a leftmost prefix of their column order. Skip the leading column, and that index has no usable entry point. In a schema like ours, where every index on every tenant-scoped table is deliberately built to lead with tenantId, that means there's nothing else for the planner to reach for either, and it falls back to a full table scan.

When our ORM joined two tenant-scoped tables, it joined on whatever its @JoinColumn metadata declared, just the structural key in this case:

@JoinColumn([{ name: 'userId', referencedColumnName: 'userId' }])

That declaration is what generates the ON clause for every find() call and every relation-based createQueryBuilder join alike: both resolve the join from the same entity metadata, so both inherited the same gap. tenantId was absent from the ON clause entirely, which meant the child table's indexes (already correctly composite, already leading with tenantId) had no usable entry point. The leftmost-prefix rule above cuts both ways: it's exactly as strict against a well-formed join missing the leading column as it is against a malformed WHERE clause.

The fix is completing that declaration. Once it’s complete, nothing further is needed.

The right-hand panel’s ON clause is doing the real work. c.tenantId = p.tenantId ties the child's tenant to the parent's, and WHERE p.tenantId = :tenantId (required on every query in this codebase) ties the parent's tenant to a literal. Postgres's planner combines the two on its own: if c.tenantId equals p.tenantId, and p.tenantId equals a specific value, then c.tenantId must equal that same value too; nobody has to write that third fact down for the planner to know it. This is called an equivalence class: a group of expressions the planner has proven are all equal to each other, so that pinning any one of them to a literal pins all of them. The planner then pushes this derived fact down as a real index condition on the child table.

In TypeORM, that’s a schema-level change: add tenantId to the same @JoinColumn array and every find() call, along with any createQueryBuilder join that references the declared relation, inherits it automatically.

@JoinColumn([
  { name: 'tenantId', referencedColumnName: 'tenantId' },
  { name: 'userId', referencedColumnName: 'userId' },
])

The ORM abstraction doesn’t disappear: entities, relations, and TypeScript types all stay.

Why Your Tests Won’t Catch It — And Neither Will Your Other Environments

We had tests. They passed. The query was fast. Nothing indicated a problem.

The join case above is closed: completing the @JoinColumn was the whole fix, and it holds at any scale. But it's worth understanding the general principle behind why a broken query can pass every check on the way to production in the first place, because it explains far more than just this one bug.

The problem starts with the data. Component tests run against a real PostgreSQL instance, but the data in that instance is whatever you inserted to set up the test: a handful of rows, sometimes just one. That’s not a dataset; it’s a scaffold. The CI seed data does better, a few thousand rows, but still orders of magnitude away from production. Whatever a query actually costs to run, indexed or not, depends entirely on how much data it has to read past, and that number is wildly different in each environment.

Here’s the part that actually explains why. When a query can’t use an index (because the index isn’t there, or because nothing in the query gives the planner a usable entry point into it), Postgres reads the table the only way left: row by row, in full. That’s not a rare fallback; it’s just what “no index” means in practice.

Reading a handful of rows this way costs nothing: a few microseconds, indistinguishable from an indexed lookup on any stopwatch a developer is actually watching. Reading millions of rows this way costs real, measurable time. The query is identically broken in both cases (same SQL, same missing index, same plan shape), but only one of them is slow enough to notice. A test environment seeded with three rows cannot tell “this query is well-indexed” apart from “this query happens to be reading a tiny table.” Both finish instantly, for entirely different reasons.

The planner does pick between several strategies for executing a join (Nested Loop, Hash Join, Merge Join) based on cost estimates that are themselves driven by these same row-count differences. But whichever strategy it settles on, a relation with no usable index still gets read in full. The strategy changes how the join is stitched together, not whether a missing index gets found.

Our environments sit at genuinely different scale tiers, and an unindexed query’s cost scales with exactly that:

A query can pass every test, look clean in staging, and run fine in several production environments, then turn into a visibly slow full table scan in the next one, once the table it’s reading unindexed finally holds enough rows to notice. Different customers use different features and accumulate different data volumes. One environment not surfacing a problem is no guarantee the next one won’t.

Automating EXPLAIN

Once we understood the problem, the fix for any individual query was clear. The process of finding them all was not. When a slow query surfaced in production, we’d pull it from logs, run EXPLAIN ANALYZE against the affected environment, read the plan, identify the issue, write a fix, and deploy. Then wait for the next one. Dozens of repositories, hundreds of query paths, multiple production environments each at a different scale tier. And every new feature added new ORM calls generating new SQL we hadn't reviewed. We needed something that ran automatically on every pull request, across every query in the codebase.

We already had the queries. Every repository had component tests running against a real PostgreSQL instance via Docker. Every query path covered by tests was executing in tests. We just weren’t looking at the plans.

We built a custom TypeORM logger that hooks into the ORM’s query pipeline. Every SQL statement gets captured: the actual raw SQL TypeORM generates, along with its bound parameters and a stack trace identifying which repository method issued it. No guessing what the ORM produces; we see it directly. At the end of the test suite, we run analysis over everything collected.

Pass 1 — Normal EXPLAIN. Run EXPLAIN FORMAT JSON on every captured query, parse the plan tree, look for sequential scans on non-trivial tables and for index scans that read significantly more rows than they return. Any such finding fails the build.

Pass 2 — EXPLAIN with Nested Loop disabled. Nested Loop has a property no other join strategy shares: because it iterates one specific outer row at a time, it can substitute that row’s actual runtime value into the inner lookup, even when nothing elsewhere in the query pins that value to a literal. Hash Join and Merge Join have no equivalent — they plan the inner side once, so a condition that’s only a correlated reference has nothing to attach to, and the inner table gets read in full. This matters because CI’s small scale would make the planner pick Nested Loop naturally anyway, hiding the fragility until production volume eventually forces a different strategy. So we run a second EXPLAIN withSET LOCAL enable_nestedloop = false, wrapped in a transaction that rolls back automatically — if index usage collapses the moment Nested Loop is off the table, that’s a real structural finding.

One caveat: for a single-row primary-key lookup, the signal is weaker, since Hash Join would never realistically get chosen there regardless of scale. The value concentrates on bulk and multi-row paths, where the outer side’s size depends on caller behavior rather than a fixed key.

Pass 3 — SQL text analysis. Some tenant isolation problems are invisible to the query plan. Some of them aren’t performance problems at all. We scan the raw SQL of every captured query for two distinct failure classes. First, and most severe: any query against a tenant-scoped table whose WHERE clause never mentions tenantId. A bare UPDATE app_assignments SET ... WHERE userId = $1 is a cross-tenant data-isolation bug, not a slow query, and this check catches it whether or not a join is involved. Second, specific to joins: ON conditions on tenant-scoped tables where tenantId is missing entirely, or present only as a correlated reference (c.tenantId = p.tenantId) rather than a bound parameter (c.tenantId = :tenantId). Text analysis can't tell whether some other part of the query independently anchors that tenant (that requires understanding the query's logic, not just its syntax), so this check flags the pattern conservatively. Pass 2 is what actually confirms whether it's fragile in practice.

Runs locally and in CI on every PR that touches the data access layer. This isn’t a separate audit step or a tool you run when you remember to. It runs as part of the existing repository test suite: automatically triggered whenever code changes in the data access layer. Locally while a developer is writing code, and in CI on every pull request. The same test that verifies what a query returns also verifies how it runs.

The key property is that the tool is inherent to how we test the data access layer. When a developer writes a new repository method, they write a component test for it. That’s standard practice. That test automatically runs the query through the logger, captures the raw SQL, and triggers the three analysis passes. There’s nothing to remember, no opt-in, no separate step. Every new query written in the system is covered from the moment its first test is written. Coverage grows with the codebase, automatically, for free.

One important detail: the CI database isn’t seeded with millions of rows, but it isn’t tiny either. Pass 1 is the one that actually needs that: a sequential scan only fails the build once the table it touches crosses a row-count threshold, since Postgres genuinely prefers a seq scan on a truly tiny table and flagging that would be a false positive. Below that threshold, a real missing-index bug could pass silently, which is why every repository spec seeds its shared tables with realistic bulk data before any test runs, rather than relying on the few rows each test inserts for its own scenario. Pass 2 and Pass 3 wouldn’t need that regardless: checking SQL text and correlation-only plans are structural properties of the query, present identically at a thousand rows or a million.

The entire system is programmatic: no AI, no inference, no external services. Pure SQL, EXPLAIN output, and text analysis. It runs completely offline, which means it works on a developer’s laptop the same way it works in CI. The results are consistent: given the same seed data, what you see locally is what CI will see.

Every finding fails the build unless explicitly acknowledged with a (kind, method, table) triple. When production code is later fixed, the acknowledgment becomes stale; the build fails again until it's removed. The record stays honest in both directions.

Taking It Further

The CI approach gets you far. But there’s a gap it can’t close: even with Pass 2 forcing the planner off Nested Loop, the test database has thousands of rows. The planner’s cost estimates (which drive index selection, parallelism decisions, memory allocation) are still based on a tiny table. Some decisions only become visible when the statistics reflect real production data.

So we took the same repository component tests and started running them once a week against a large-scale synthetic database that mirrors our production data volumes. No simulation, no tricks: just the actual planner operating on actual statistics. A query that passed every CI check at small scale runs against tens of millions of rows and either holds up or doesn’t. Structural warnings from Pass 2 get confirmed or cleared. New failure modes that only emerge at real scale appear before they reach production.

This gives three tiers of coverage:

• Every PR — small seed data, structural detection via the no-Nested-Loop fragility check
• Weekly — production-scale database, real planner decisions at real statistics
• Production monitoring — slow queries detected at runtime, EXPLAIN run automatically, ticket opened with the query, its EXPLAIN plan, and the violation type

The on-call engineer gets context, not just an alert: the query, its EXPLAIN output, and a structured fix recommendation. The developer gets a failed build before the code merges. The weekly run catches everything in between.

The tool that started as a CI gate became a full coverage loop — from the first keystroke to production.

The Takeaway

The ORM writes correct SQL, not necessarily efficient SQL at scale. It has no opinion on your indexes, no awareness of your row counts, and no idea what the query planner will decide when the table grows. That gap is invisible by design. And invisible gaps don’t show up until something breaks.

The real problem isn’t any single bad query. It’s that there was no mechanism to know. Queries shipped, environments scaled, the cost became visible — and we found out in production. A one-time audit doesn’t fix that. You need a system that runs on every PR, covers every query path in your codebase, and enforces standards going forward automatically, so coverage grows with the code, not with the incidents.

The queries were always there. We just needed to start reading them.

The ORM Trap: How We Started Catching Database Problems Before Production was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Bartosz Szafran & Adi Dimant
Engineering Team, Transmit Security

Fraudsters often reuse resources that legitimate users rarely share — different accounts logging in from the same device or different devices connecting from the same IP address. When multiple overlaps occur, they form networks that may signal coordinated fraud.

At Transmit Security, we built a system to identify these networks. In this post, we’ll explore how we detect these connections, filter out noise to focus on the most suspicious patterns and present the results as an interactive graph. The article is divided into three parts: problem statement, detection and interface.

Problem statement

Our systems monitor and collect user events as they interact with protected applications. In each user action, we may have a range of features and attributes — from user account data and device identifiers to behavioral signals like keyboard typing speed. Given a set of these actions, we would like to correlate them to identify coordinated fraud.

The result is what we call a fraud ring — a network of entities connected through shared attributes, visualized as an interactive graph.

Detection

The detection of fraud rings is a multi-stage process. Raw action data is transformed into a graph structure, where connected components reveal clusters of related entities. Let’s walk through each stage of the detection.

Pipeline overview

Fraud ring detection runs as a batch pipeline at regular intervals. In each run, the system processes new actions and merges them with historical data to get a continuous view of fraud activity. The pipeline is written in Python based on Apache Beam, and relies on NetworkX for graph and network analysis. The full graph is processed in-memory in each run, and persisted to disk across runs.

The pipeline follows a two-stream architecture:

Input phase

1. Accumulator stream — fraud rings detected in a previous run, loaded from file storage
2. Actions stream — actions from the last batch, queried from the database

Detection phase

1. Both streams are turned into graphs
2. Both graphs are merged (edge weights are summed, per-node action counts accumulate)
3. Connected components are extracted from the merged graph. Each connected component represents a potential fraud ring
4. Components are filtered to remove noise
5. Each surviving connected component is assigned a fraud ring ID

Output phase

1. The filtered components are saved as the new accumulator
2. Aggregations (per-entity statistics) are published to our ML detection models
3. Alerts are sent for significant new fraud rings

This architecture allows fraud rings to grow over time — each run builds on the accumulated history.

Building the graph

The first step in detecting a fraud ring is constructing a graph that represents the relationships between entities. This graph becomes the foundation for all analysis.

The system queries an actions database to get all actions from the latest time interval. Each action includes a set of attributes. These attributes define the nodes in the graph, and their occurrence in multiple actions creates edges between them. The construction process works as follows:

1. Node creation — for each unique entity found in the action attributes.
2. Edge creation — for each action, we create edges between every pair of entities present in that action.
3. Edge weighting — each edge has a weight representing how many actions created that connection. For example, if 2 attributes appeared together in 5 actions, their edge weight is 5. Higher weights indicate stronger relationships between entities.

This process produces a graph where connected components represent potential fraud rings. A single detection run may identify multiple disconnected clusters, each creating a separate candidate.

Example: from actions to graph

Let’s walk through an example. Suppose we have four actions:

Each action’s attribute value becomes an entity in the graph. Then the action itself creates connections between these four entities. From the actions presented in the table, we extract:

Nodes (9 total):

• 3 names: Sarah, John, Emma
• 3 home addresses: Oak Lane, Riverside Plaza, Maple Street
• 3 types of car: silver sedan, blue hatchback, red SUV

Edges (up to 12 unique edges, 3 per action — repeated connections increase weight):

• Action 1: Sarah — Oak Lane, Sarah — silver sedan, Oak Lane — silver sedan
• Action 2: John — Riverside Plaza, John — silver sedan, Riverside Plaza — silver sedan
• Action 3: John — Oak Lane, John — blue hatchback, Oak Lane — blue hatchback
• Action 4: Emma — Maple Street, Emma — red SUV, Maple Street — red SUV

The resulting graph reveals the suspicious pattern:

Notice how the silver sedan connects Sarah and John — this is a potential red flag. Additionally, Oak Lane is shared between Sarah (via Action 1) and John (via Action 3). These shared resources are what link them into a fraud ring.

This way, we’ve built a graph that links entities through shared attributes observed in the same actions — previously unknown relationships now seen as connections in the structure. The next step is to extract the connected components.

Extracting connected components

To find connected components, we call networkx.connected_components, which runs the BFS algorithm. Two nodes end up in the same component only if there’s a path between them.

From this point on, the pipeline stops working on one graph and starts working on connected components. Filtering, ID assignment, and storage all run per-component, independently and in parallel.

Filtering components

Once connected components are identified, they’re filtered to focus on real threats. Not every cluster of connected entities represents fraud — some are noise, some are legitimate shared infrastructure. The filtering process separates signal from noise.

The system filters components based on various criteria, such as:

Cluster size

Very large clusters typically indicate legitimate shared infrastructure (such as a generic public IP), where high connectivity represents noise rather than coordinated fraud. On the other hand, very small clusters often represent legitimate relationships — for example, two users sharing the same Wi-Fi network.

To address this, the system applies a maximum size threshold (max_size_threshold) unconditionally, and filters clusters below min_size_threshold only when their earliest activity is before cutoff. New small clusters are kept so they can grow on later runs.

Node trust rate

Each node may be associated with multiple actions in our system. The system evaluates these actions and establishes a trust level for them using various methods, such as heuristics and AI models. These trust levels are then combined to calculate an overall trust level of the node.

If more than trust_rate_threshold% of entities in a cluster have been trusted, the cluster is filtered out. A high trust rate typically indicates known legitimate users.

Activity age

Inactive clusters may have been abandoned or already addressed. Components with no new activity for more than retention_threshold days are filtered out.

Persisting and restoring the accumulator

The pipeline needs to preserve the detected fraud rings between runs. We write them to a file storage as the accumulator — the same artifact that feeds the “Accumulated components from previous run” input in the pipeline diagram.

We had two options for the on-disk data structure:

• one big graph object containing every component,
• each connected component is saved as its own self-contained graph.

Two fraud rings never share an edge, because any connection between them would merge them into one. That means both forms would contain exactly the same information.

We chose the second option because the pipeline already handles components individually. Every per-component step — filtering, ID assignment, writing — runs independently, since components share no state. Storing one component per record matches this structure directly, without an intermediate step to merge everything into one big graph object again.

Note that we store only components that passed the filter — everything else is dropped and won’t appear in the next run.

The next run picks up from the accumulator. Before processing new actions, the pipeline reads the accumulator records in parallel and decodes each line back into a single-component graph. Together they form an in-memory universe of fraud rings carried over from previous runs. Because components never share an edge, building the universe is straightforward: each loaded component is added to a shared graph object, sitting alongside the others without overlap.

The universe is then merged with the graph built from new actions. From there the cycle repeats: connected components are re-extracted from the merged graph, filtered, and stored again as the next accumulator.

Tracking fraud rings across runs

Fraud rings evolve between runs. From the system’s perspective, every component coming out of a run falls into one of three cases:

1. A brand new ring — it didn’t exist in the previous accumulator.
2. An existing ring that grew — new actions added entities or strengthened existing connections, but the ring is still the same ring.
3. A merger of two or more existing rings — a new shared entity has connected previously separate rings, collapsing them into one.

To make this evolution useful (tracking growth, updating the UI, avoiding duplicate notifications) we need to recognize the same ring across runs. We assign every fraud ring a unique ID and attach it to every node in the ring. The ID is part of the components data we write to the accumulator, so it’s recovered when the next run loads the universe. This gives each node two pieces of identity: the entity itself (a name, a home address, a type of car) and the ID of the ring it currently belongs to.

The three cases translate directly into how IDs are assigned. After extracting and filtering each component, the system looks at the IDs already attached to its nodes:

1. Brand new ring — none of the nodes carry an ID. We generate a fresh UUID and attach it to every node.
2. Existing ring grows — every node in the component has either the same ID or no ID. We keep that ID and attach it to any nodes without one.
3. Rings merge — nodes carry two or more different IDs. We pick the ID held by the most nodes (effectively the largest of the pre-merge rings) and overwrite every node in the component with it. The other IDs are dropped.

User interface

Fraud rings can grow large, containing thousands of nodes. Visualizing them is essential to investigate and act on threats. The most significant fraud rings are promoted to the Mosaic Portal, where users are notified and can investigate.

Rendering at scale: two decisions

Displaying a graph of thousands of nodes in the browser required two separate decisions:

1. Layout — positions for each node must come from somewhere. We can compute them ahead of time on the backend, in a web worker, off-screen on the main thread, or live as the simulation runs.
2. Rendering — how to draw. The graph can be rendered as HTML/SVG DOM elements, on a 2D canvas, or through WebGL.

Layout: off-screen force simulation

We use D3’s force simulation to automatically arrange nodes. The algorithm works in an iterative way, revealing the network structure. It’s a physics-based approach where nodes push each other apart while edges pull connected nodes together.

We considered a few ways to run this computation:

• Live-animated layout — re-render the graph on every simulation tick. The animation is visible to the user, but for a graph with hundreds of nodes and hundreds of iterations, every tick triggers a full re-render. The browser can’t keep up and the UI starts to lag.
• Off-screen, main thread — run the simulation in memory. We draw the result only when the calculation ends, so there’s exactly one commit regardless of iteration count. The user sees no graph until the simulation finishes, but because the total number of iterations is known, progress is deterministic and we can display a progress bar until the final layout is ready.
• Web Worker — move the simulation off the main thread. It could be valuable for massive graphs, but at a scale of a few thousand nodes, the off-screen main-thread calculation takes less than a few seconds. The simulation schedules each iteration separately, so the main thread stays responsive between them. The extra complexity isn’t justified.
• Server-side precomputation — store layout coordinates with each ring. We rejected this because layout parameters are UI-specific concerns and may vary, for instance, depending on the viewport size.

We went with the off-screen, main-thread approach — a good trade-off between implementation effort and rendering performance.

Our simulation combines three forces:

• Center force — pulls all nodes toward the center, preventing the network from growing too wide.
• Charge force — makes nodes push each other apart, creating space between them.
• Link force — pulls the nodes closer based on their edges.

As a result, densely connected entities show up together and loosely connected groups spread apart. This makes it easier to spot suspicious patterns visually.

Rendering: HTML/SVG over Canvas and WebGL

Our graph has specific interactivity requirements: each node has a tooltip with details, a highlighted border when selected, and click handlers that open the details side panel. Fraud rings contain at most a few thousand nodes.

We considered three rendering approaches:

• HTML/SVG — each node and edge is a DOM element. The interactions are straightforward to implement, since the browser maps clicks and hovers to the right node automatically. The trade-off is performance — DOM trees start to lag at tens of thousands of nodes.
• 2D Canvas — draw the entire graph into a single canvas element. Rendering scales to tens of thousands of nodes, but the canvas is one opaque pixel buffer to the browser — detecting which node the pointer is on, hover effects, and tooltips have to be implemented manually.
• WebGL — offload rendering to the GPU. Performance is the highest, easily handling hundreds of thousands of nodes, but it’s the most complex solution. As with canvas, all interactivity has to be implemented manually.

At our scale, HTML/SVG performance is sufficient and interactivity wins the trade-off. We chose React Flow — an HTML/SVG-based graph library — rather than building one ourselves.

Off-screen layout has already removed the per-tick render cost, leaving just a single commit to render.

Floating edges: skipping the trigonometry

The default way of connecting edges to nodes in React Flow is through anchors at specific points of the node — right, top, bottom, or left. This works well for rectangular nodes. Fraud ring nodes, however, are circular and can receive connections from any direction.

React Flow’s official floating edges example solves this with trigonometry: for each edge, compute where the line intersects each node’s circular perimeter. We use a simpler approach — draw each edge directly from the center of one node to the center of the other. The opaque circular nodes cover the line endpoints, so the result looks the same as a proper perimeter intersection. The shortcut is simpler to implement and faster to render.

Summary

The fraud ring detection system builds graphs from attributes collected by our systems, linking accounts that share the same resources. By filtering out noise and evolving over time, it becomes an effective tool for identifying and visualizing suspicious patterns — making coordinated fraud easier to detect.

Detecting Coordinated Fraud: From Shared Attributes to Interactive Graphs was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Engineering Team, Transmit Security

This is the second post in our engineering blog series. In this post, we’ll explore the engineering behind real-time document and selfie capture, from on-device ML models to cross-platform challenges and the architecture that ties it all together.

You’ve probably done it a dozen times: pointed your phone camera at an ID card, tapped the shutter button, and… blurry. Glare. Half the card cut off. Try again.

Auto Capture exists to kill that frustration. It guides users in real time and automatically captures clear images of documents and selfies, no manual shutter tap required. But making that feel effortless took an on-device ML pipeline, platform-specific inference runtimes, backend-driven model selection, and enough edge cases to keep an engineering team caffeinated for months. Let’s break it all down.

Two Flows, One Goal: Understanding What Auto Capture Actually Does

Before we dive into the pipeline, it’s worth understanding that Auto Capture isn’t a single monolithic system. It’s two distinct flows that share a common goal: capturing the best possible image without burdening the user.

Both flows answer the same fundamental questions for each camera frame: Is the relevant object present, a document or a face? Is it properly positioned? And is the image suitable for downstream identity verification, whether that means checking for blur, glare, face rotation, or profile angle?

Document Auto Capture gates capture readiness for the front and back of ID cards and driver’s licenses, or just the front for documents like passports. At its core, on-device ML models handle two jobs: a document detector locates and validates the document in the frame, while a dedicated classifier evaluates image quality for blur and glare in real time. Only when both checks pass is capture allowed. Once captured, the image is sent to the backend where the real orchestration begins. For front-side captures, the backend runs document classification, identifying the document type and determining whether a back-side capture is needed (and whether that back is expected to carry a barcode). If it is, the SDK transitions the user into a back-side capture flow, running through the same on-device quality gate before sending the image for backend processing, including classification matching and barcode decoding. To the user, it’s one smooth flow. Behind the scenes, the backend is driving a multi-step acquisition pipeline, but it’s the on-device capture gate, running on every frame for both sides, that makes the experience feel instant and intelligent.

Selfie Auto Capture is a separate flow built around face detection with quality and composition checks. Rather than image quality classification, it gates capture on face presence, pose, positioning, and occlusion. It carries its own set of real-world challenges: people blink, move, wear sunglasses, and generally make the camera’s job harder than a flat card ever would.

Both flows evaluate frames on-device, giving users real-time feedback without a round trip to the server, keeping latency low and the experience smooth.

Model Compression: Fitting ML on a Mobile Device

Before we get into the specifics of each pipeline, let’s talk about the elephant in the room, or rather, the elephant you’re trying to fit into a phone. Running ML models on-device means working within strict constraints: limited memory, limited compute, and the expectation of real-time performance. Server-grade models that work beautifully in the cloud would melt a phone’s battery and patience.

Our approach starts at the model architecture level. Rather than simply shrinking server-grade models after the fact, we develop and select architectures that are fundamentally designed for mobile inference, lightweight by nature, not just by compression. This means choosing model families and structures that deliver strong accuracy within the compute and memory budgets of mobile devices from the ground up.

On top of that, we apply quantization, reducing model precision from 32-bit floating point to lower-precision representations, which further shrinks model size and accelerates inference with minimal accuracy loss. The trade-off is subtle: quantization artifacts can appear in edge cases, so each quantized model goes through rigorous validation against the full-precision baseline.

Each target platform also has its own export and conversion pipeline (Core ML, TFLite, TensorFlow.js), and each conversion step is an opportunity for runtime-specific optimization. Operator fusion, graph simplification, and format-specific tuning all contribute to squeezing out additional performance during the export process.

As we develop new model versions, each goes through this export and validation pipeline per platform, ensuring the optimized models meet quality thresholds on every target runtime before they reach a single device.

Document Auto Capture: The On-Device Quality Gate

The document Auto Capture pipeline is, at its heart, a quality gate. The device continuously streams camera frames, and for each frame, the system asks one critical question: is this image good enough to capture?

Finding the Document First: The Document Detector

Before we can judge image quality, we need to know there’s actually a document to judge. The pipeline starts with a document detector that locates the document within each camera frame, identifying its boundaries and confirming it’s sufficiently visible and properly framed.

This might sound trivial (it’s a card, how hard can it be?), but users have a remarkable talent for creative camera angles, partial framing, and holding documents at distances that would challenge even the most advanced ML model. The detector provides the foundation that everything else builds on: no detected document, no point evaluating blur or glare. It also feeds real-time guidance back to the user (“move the document into frame,” “hold it closer”) so they know what to fix before the quality checks even kick in.

The Blur/Glare Classifier: Gatekeeper of Image Quality

Once the detector confirms a document is present and properly framed, the primary capture decision runs through a MobileNetV2-based classifier purpose-built for the Auto Capture use case. For every candidate frame, this model evaluates the image and outputs one of several classes: success (good quality), blur, glare, or bad-quality.

If the image passes, capture is triggered. If it doesn’t, the frame is rejected, the user receives real-time guidance (“hold steady,” “move to better lighting,” etc.), and the device keeps scanning. This loop runs continuously until a quality frame is captured, which to the user feels like the camera just “knows” when the shot is right.

Why is this a dedicated ML model rather than a simple heuristic? Because blur and glare are surprisingly context-dependent. A Laplacian variance check might work in a lab, but real-world conditions, variable lighting, moving hands, glossy vs. matte card surfaces, demand a learned classifier that generalizes across environments. The model was trained on a wide range of real-world capture conditions, which gives it robustness that heuristics with hand-tuned thresholds simply can’t match.

The classifier’s thresholds are configurable, which becomes important when we talk about backend-driven configuration later. What counts as “too blurry” isn’t a universal constant. It depends on the downstream processing requirements, the document type, and the capture environment.

After Capture: Backend Acquisition

Once the on-device quality gate approves a frame, the image is sent to the backend for acquisition, heavier processing that doesn’t need to run at camera frame rates. The backend handles document classification, determines whether a back-side capture is needed, and runs barcode decoding to extract structured data from formats like PDF417 on driver’s license backs. It can also perform alignment and refinement, including segmentation-based background removal and perspective correction.

This split between on-device capture gating and backend acquisition is a key architectural decision: keep the device focused on one fast operation (quality classification), and let the server handle the rest.

Selfie Auto Capture: Composition Over Classification

Selfie Auto Capture tackles the same core questions (object present, properly positioned, suitable for verification) but through a different lens (pun mildly intended). Where document capture centers on image quality classification, selfie capture focuses on face quality and composition: Is there a face? Is it properly framed? Will it hold up for identity verification?

The Face Detection Foundation

Every frame starts with face detection. No face found? Rejected. Multiple meaningful faces detected (when that check is enabled)? Also rejected: the system needs exactly one clearly visible face. When a valid detection comes through, the pipeline evaluates the largest face candidate and moves on to the quality checks.

Fast Geometric and Pose Checks

From the detector output, bounding box and landmarks, the pipeline runs a series of lightweight checks that provide immediate capture guidance:

Geometry and positioning checks ensure the face isn’t too close, too small, too close to the image borders, or cropped. These are derived directly from the bounding box and landmark geometry that the face detector already provides, making them computationally cheap to evaluate.

Pose checks reject frames where the face is rotated or angled beyond acceptable thresholds. Identity verification needs a reasonably front-facing capture, and catching bad angles at capture time saves users from a rejection further downstream. Nobody wants to learn their selfie was unusable after they’ve already waited for a server round-trip.

These checks are deliberately fast and heuristic-based. Face detection gives us rich enough geometry to derive these quality signals without additional inference passes, which is exactly what you want when you’re evaluating frames at camera speed on a mobile device.

Server-Side Model-Backed Quality Checks

Some conditions can’t be caught by bounding box math alone. Once a frame passes the on-device geometric checks and reaches the backend, dedicated neural network models handle the trickier quality conditions:

Closed eyes detection uses a model to determine whether the user’s eyes are shut. Squinting, heavy eyelids, and the wide variation of eye shapes across populations make this a problem that genuinely benefits from a learned classifier rather than a geometric heuristic.

Face occlusion detection uses a classifier to catch obstructions: sunglasses, hands, masks, or a particularly enthusiastic scarf. Rather than trying to enumerate every possible occlusion through rules (a losing game given the creativity of real-world users), the model classifies whether the face is sufficiently unobstructed for identity verification.

Running these checks server-side is a deliberate architectural choice. The models benefit from more compute headroom than a mobile device can comfortably offer, and keeping them on the backend means we can update models and tune thresholds without pushing a new app release.

The combination of fast on-device geometric filtering and server-side model-backed checks creates an autocapture gate that catches problems at two stages, giving users instant feedback on positioning while quietly ensuring the deeper quality conditions are met before a selfie moves on to identity verification.

Cross-Platform Challenges: One Pipeline, Three Worlds

Here’s where things get spicy. Auto Capture needs to run on iOS, Android, and the Web. Same user experience, same quality bar, but completely different runtime environments under the hood.

iOS: Core ML

Apple’s Core ML framework provides tight integration with the Neural Engine and GPU on iOS devices. Models are converted to Core ML format with quantization support, enabling hardware-accelerated inference optimized for Apple’s silicon. The upside: excellent performance on modern iPhones and iPads. The challenge: Core ML has its own set of operator support limitations and quantization behaviors, so model conversion isn’t always a straightforward export.

Android: TensorFlow Lite

On Android, models are exported to TensorFlow Lite, which can leverage device-specific acceleration paths depending on the hardware and integration. TFLite’s flexibility is essential given the enormous range of Android devices, from flagships to budget phones that are, charitably, doing their best. The diversity of the Android ecosystem means more testing, more edge cases, and more creative optimization to ensure consistent behavior across device classes.

Web: TensorFlow.js with WebAssembly Backend

On the Web, models are delivered via TensorFlow.js, with WebAssembly backend assets available for CPU-oriented execution in-browser. While WebGL and WebGPU can provide some acceleration in certain browser contexts, the WASM path offers the most consistent cross-browser behavior.

This means the Web models need to be the leanest variants, and inference must be carefully optimized to maintain real-time frame rates without turning the user’s browser tab into a space heater.

The Conversion Tax

Each platform’s runtime expects models in its own format, which means maintaining conversion pipelines from the base trained model to Core ML, TFLite, and WASM-compatible formats. Each conversion introduces its own quirks: operator compatibility, numerical precision differences, and performance characteristics that need to be validated per platform. It’s not glamorous work, but it’s the kind of infrastructure that makes or breaks a cross-platform ML product.

Dynamic Model Loading: The Right Model for the Right Device

Not all devices are created equal, and pretending otherwise leads to either poor performance on low-end devices or wasted capability on high-end ones. We solve this with a backend-driven dynamic model loading and caching pipeline.

Here’s how it works: the backend maintains a device profile mapping that associates device characteristics with the most suitable model variant. When the SDK initializes, it reports its device profile, and the backend responds with the appropriate model configuration, selecting the variant that balances accuracy and performance for that specific device class.

Models are downloaded and cached locally, so subsequent sessions don’t pay the download cost again. If a new model variant is available (say, we’ve shipped an improved blur/glare classifier), the backend can push the update without requiring a new app release.

This architecture gives us several powerful capabilities:

Silent updates. Model improvements, threshold adjustments, and even new model architectures can be deployed without going through app store review cycles.

A/B testing. We can roll out model variants to subsets of devices and measure real-world performance before a full rollout.

Backend-Driven Configuration: Tuning Without Shipping

Beyond model selection, the backend also controls runtime configuration for the capture pipeline. This includes thresholds for blur and glare sensitivity, retry logic parameters, capture timeouts, and other behavioral settings.

Why does this matter? Because the “right” threshold isn’t universal. Each customer operates in a different environment: some deployments are predominantly indoor (think bank branches or office onboarding), while others are outdoor or mixed (field agents, remote identity verification on a park bench). A blur threshold tuned for controlled indoor lighting might be too aggressive for outdoor captures with natural lighting variation. A glare threshold that works well for matte ID cards might reject too many frames when users have glossy laminated documents.

This is configuration as a feature, not a workaround. Different customers, document types, and capture environments have different sweet spots, and those sweet spots can shift over time. Backend-driven configuration lets us continuously optimize the capture experience based on real-world performance data, adapting thresholds and retry logic to each deployment’s specific needs without requiring an app update.

Observability: You Can’t Improve What You Can’t Measure

Auto Capture emits detailed metrics at each stage of the pipeline. These events give us visibility into model performance and user behavior in production, not just in controlled test environments.

We track signals across the capture lifecycle: quality gate pass/fail rates, blur and glare rejection frequency, selfie composition rejection reasons, overall capture success rates, and user interaction patterns like how many attempts it takes to get a successful capture.

This data feeds a continuous improvement loop. It tells us where users are struggling, which device classes underperform, whether a model update improved real-world outcomes, and where our thresholds need adjustment. Without this observability, we’d be flying blind, shipping models into production and hoping for the best.

What Makes It All Click

Auto Capture might look simple from the outside: point your camera, get a good image. But underneath that simplicity is a carefully designed system where an on-device quality gate, platform-native inference runtimes, backend-driven intelligence, and server-side acquisition processing each play their part.

The architectural split between on-device and backend is perhaps the most important decision in the whole system. By keeping the device-side pipeline focused and fast (a quality classifier for documents, a composition checker for selfies), we deliver the real-time responsiveness users expect. By handling the heavier lifting, classification, alignment, barcode decoding, on the backend, we get access to more compute, easier updates, and a cleaner separation of concerns.

The other engineering decisions, using platform-native inference runtimes, delegating model selection to the backend, keeping configuration server-driven, and building deep observability, aren’t accidental either. Each one reflects a real lesson learned from operating ML at scale across a fragmented device ecosystem.

We’ll continue exploring more of our engineering challenges in future posts. Stay tuned.

Gal Yona and Ophir Yaniv are engineers at Transmit Security, where they work on the identity verification platform that processes millions of document and selfie captures across the globe.

Inside Auto Capture: How We Engineered Real-Time Document and Selfie Capture Across Every Platform was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Architecting for Scale

Decoupling Session Management in FlexID: Architecture Evolution and Operational Insights

Written By: Alex Brus and Ido Kislev

Introduction

In authentication systems, session management is critical infrastructure: sessions track who is authenticated, from which device, and maintain security context throughout a user’s interaction. Few years ago, as we were helping a customer to deal with a particularly high-scale deployment, it became apparent that the FlexID session storage architecture had a fundamental coupling between user- and session-data that limited the system’s ability to scale.

And so, what started as a performance investigation became a comprehensive refactoring effort that spanned a significant part of the system’s codebase, involving foundational changes to the codebase to introduce a context-based architecture that enables decoupling of session and user storage. This in turns allows independent scaling, flexible storage backends, and zero-downtime migration.

This post details the journey of architecturally separating session storage from user storage in FlexID. Developers and architects will find technical insights into executing large-scale architectural refactoring in a mission-critical system, while maintaining backward compatibility and zero downtime.

The Problem: When Tight Coupling Becomes a Performance Tax

FlexID was built around a document store as its primary OLTP data store. When modeling the data store we originally followed paradigm that’s commonly applied for document stores: store data that is used together in the same document. Hence, sessions were embedded directly within user documents. From a data modeling perspective this made sense: sessions belong to users, so why not store them together?

{
  "User": {
    "id": "string",
    "email": "string",
    "applications": [
      {
        "Application": {}
      }
    ],
    "devices": [
      {
        "id": "uuid",
        "hardwareId": "string",
        "signingKey": {
          "RotationKey": {}
        },
        "applicationId": {
          "ApplicationId": {}
        },
        "sessionOpt": {
          "Session": {}
        }
      }
    ]
  }
}

In fact the structure was a little more sophisticated: rather than containing sessions directly within user documents, users contained devices and devices contained sessions. Every authentication request meant loading the full user document, extracting the relevant device, and then reading the embedded session.

The Pain Points

The first challenge was the monolithic data model. Session data lived embedded within user documents, which meant that every session operation, whether a simple validation or a minor update, required loading and writing the entire user record. This led directly to inefficiency: sessions could not be queried independently of users, and their lifecycles became unnecessarily coupled. A session expiring should not require updates to user data, but the FlexID architecture made this unavoidable. Similarly, session validation required loading the entire user document just to check if a token was valid. Simple session updates, such as updating a “last accessed” timestamp, triggered writes to the full user record. Operations that only needed session data implied the cost of loading complete user profiles with their full device history.

FlexID utilizes a stateless architecture to facilitate significant horizontal scaling. The stateless architecture compounded the problem: every request scoped to a user session (this encompasses most requests to an authentication server, obviously) required loading the session data from the database, and typically also writing it back. Again, as sessions were embedded within user documents, this meant loading/storing the entire user object including all devices and their history on every request.

It’s important to note that in the original FlexID data modelling, session data comprises a significant portion of the user document. As sessions were created, updated, and expired, the user document would expand and contract significantly causing unpredictable document size fluctuations that complicated database operations and caching strategies. Moreover, completed or expired sessions would be cleaned up asynchoronously to their expiration, leading to a “dead” session accumulation problem. This issue was particularly acute for on-premise deployments, due to Transmit’s lack of direct control over the customer configuration or background processes. User cleanup settings were optional, and many customers either didn’t enable them, or only configured them partially. The result? Pending sessions from failed or abandoned authentications remained in user documents indefinitely. Expired sessions stayed in records for months or even years. Some user documents accumulated dozens or even hundreds of dead sessions, causing document sizes to grow unbounded and degrading all database operations reads, writes, sharding, and replication. While this problem is largely obsolete in FlexID SaaS deployments with automatic cleanup processes, on-premise customers with high-scale deployments were potentially significantly impacted.

Finally, with the original architecture session storage could not be based on a different data store than user storage. Session data and user data are fundamentally different. Session data is highly volatile whereas much of the user data is immutable and the rest is only seldom updated. With the architecture forcing a unified data store for both, the session’s data store could not be independently scaled, nor could it be optimized for ephemeral data. Multi-region deployments were also constrained by unified storage requirements.

The Breaking Point

Under load testing simulating some of our largest enterprise customers’ traffic patterns, the operational database reached 60% CPU utilization on the hottest shard. Session operations consumed resources meant for user operations, and there was limited headroom for traffic spikes.

The Solution: Architectural Separation

Moving forward required a fundamental architectural change rather than incremental fixes. It was crucial to treat sessions and users as entirely independent domain objects with no embedded references, no shared storage, and no coupled lifecycles.

The solution was based on four principles:

1. Independent domain models: Sessions and users would become entirely separate objects with no direct references between the objects. Each object could be optimized for their own access patterns without compromising the other.
2. Context-based orchestration: Rather than embedding relationships in the data model, these would be managed through context objects at runtime — load a user from one store, load a session from another, and combine them in a context object for business logic. The storage layer stays clean and the orchestration layer handles complexity.
3. Dual-mode operation: Both legacy and new modes would be supported simultaneously. Configuration-driven switching would enable zero-downtime migration. Simply flip a flag to change storage strategy without code deployment.
4. Database flexibility: Session storage would be based on a pluggable interface using MongoDB initially, then Redis, and also supporting PostgreSQL for deployments with specific datastore requirements.

The Session Repository Interface

Sessions are written under a globally unique sessionId. In the repository API we always pass userId alongside it to scope access and validate ownership. For device-bound flows we also support lookup by (userId, deviceId) to retrieve the current active session for that device and enforce the one-session-per-device constraint.

A clean repository abstraction supporting multiple access patterns was defined. The interface needed to handle lookups by session ID, user ID, and device ID, with each optimized for its specific use case.

trait SessionRepository {
  // Primary operations
  def findById(userId: UserId, sessionId: UserSessionId): Future[Option[UserSession]]
  def findByUserDeviceId(userId: UserId, deviceId: UUID): Future[Option[UserSession]]
  def findByUserId(userId: UserId): Future[Seq[UserSession]]
  
  // Lifecycle operations
  def save(userSession: UserSession): Future[SaveResult]
  def upsertByUserDeviceId(userSession: UserSession, deviceId: UUID): Future[SaveResult]
  def delete(userId: UserId, sessionId: UserSessionId): Future[SaveResult]
}

How we do (userId, deviceId) lookup in Redis (concrete, matches code)

In the Redis implementation we keep two keys:

• Session record: sessions:<sessionId> → JSON payload (with TTL)
• Device index: user_device_index:<userId>:<deviceId> → <sessionId> (with same TTL)

On write, we update the session record and the index in one atomic batch, and if the device index already points to a different sessionId we delete the old session (replace behavior to enforce the constraint).

The New Session Model

The session model captures all necessary metadata for independent operation. Key design decisions included optional device binding (supporting both device-bound and web/API sessions), embedded TTL for database-native expiration, realm awareness for multi-tenancy isolation, and session ID as the primary key for direct access without compound keys.

case class UserSession(
  session: Session,              // The actual session data
  realm: Realm,                  // Multi-tenancy context
  userId: UserId,                // User association (reference only)
  deviceId: Option[UUID] = None, // Optional device binding
  ttlDatum: DateTime             // Expiration timestamp for DB-native TTL
)

Consistency Considerations

Separating storage introduces a critical question: what happens when session data and user data diverge?

Our Consistency Model:

Separating session storage introduces a natural question: what happens when session data and user data temporarily diverge? In the embedded model, the user document and the session were updated together as a single record. Once sessions live in a dedicated store, the two objects can momentarily fall out of sync (e.g., a user/device change is committed while a session update is still in flight, or a session lingers briefly after a user is deleted). We address this with an explicit, pragmatic consistency model that matches the reality of authentication traffic: sessions are ephemeral, and correctness is maintained through scoping, TTL, and deterministic convergence rules.

Our consistency model

• Sessions are eventually consistent with user state: session reads are always scoped and validated (by userId, and when relevant applicationId/deviceId). If the session doesn’t match the expected scope, we treat it as missing and the client re-authenticates.
• Session operations don’t require user locks: routine session reads/writes touch only the session record, avoiding contention and lost updates on the user document.
• User deletions trigger async session cleanup: deleting a user schedules/executes deletion of that user’s sessions. A small lag is acceptable (e.g., a few seconds) because sessions are validated and time-bounded.
• Device binding uses optimistic concurrency: for device-bound sessions we enforce “one active session per (user, device)” at the session store level (unique constraint/index where deviceId exists) and rely on upsert semantics so concurrent attempts converge to a single outcome.

Edge case: concurrent login from the same device

If a user starts two authentications from the same device at the same time, both requests may attempt to create a device-bound session. The session store enforces the single-session-per-device constraint, so one attempt wins and the other retries using an upsert, resulting in a single active session for that (userId, deviceId) pair. This “last writer wins” convergence is acceptable here because these are journey sessions: by the time the flow completes, the system has converged to one canonical session and the client proceeds with that state.

Preventing Accidental Session Writes

In this new mode, sessions should never be written to device objects. Runtime guards were added that throw exceptions if code attempts to write sessions to the wrong storage. These checks caught several edge cases during development that would have caused subtle bugs in production.

Session Constraint Enforcement

In the legacy architecture, the one-session-per-device constraint was enforced by document structure and you simply couldn’t have two sessions in the same device object. With separated storage, this needed to be enforced programmatically:

• MongoDB: Unique compound index with partial filter.
• PostgreSQL: Unique constraint on (user_id, device_id) where device_id is not null.
• Redis: In Redis, we maintain a primary key per session (sessions:<sessionId>) and a secondary index mapping each (userId, deviceId) to the current session ID. Writes update the session record and the index together in a single atomic batch, so we never observe a partial state where the index points to a non-existent session (or vice versa).

Choosing a Storage Backend

Different deployment scenarios call for different storage backends. Three storage providers were implemented for session data, each optimized for a specific use case. The pluggable architecture means organizations can select and switch between backends based on their specific requirements in on-prem deployments, while for a SaaS deployment we apply the suitable solution.

Dual-Mode Architecture: Zero-Downtime Migration

Our new architecture supports two operational modes, giving you the flexibility to choose the approach that best fits your deployment requirements. Legacy mode keeps sessions embedded in user documents, while the default mode stores sessions in dedicated storage. This dual-mode design enables zero-downtime migration and instant rollback capability.

Business logic operates on context objects, completely unaware of which storage mode is active; hence the switch is transparent to all calling code.

Migration Strategy: Four-Phase Rollout

Phase 1 Dark Launch
Deploy code with legacy mode. New infrastructure deployed but dormant. Validate dual-mode code paths work.

# All environments initially transient-database: sessions: enabled: false # Legacy mode active

Phase 2 Internal Validation
Enable in dev/staging environments. Run full integration tests in both modes. Validate all session operations. Fix edge cases discovered.
Validation:

• All integration tests pass in new mode
• Performance benchmarks confirm improvements
• Session operations function identically

Phase 3 Gradual Production Rollout Enable for subset of production traffic. Monitor metrics at each stage. Gradual increase: 25% → 50% → 75% → 100%.

# Environment-specific configuration production-cluster-1: transient-database.sessions.enabled: true # 25% traffic production-cluster-2: transient-database.sessions.enabled: false # 75% traffic

Monitoring:

• Session operation metrics
• Error rates comparison
• Performance monitoring
• User experience validation

Instant Rollback

Rollback can mean two things: (1) reverting the storage mode, or (2) reverting the deployed version. The preferred rollback is reverting to legacy mode using the same dual-mode binary (configuration change + rollout), which is immediate and does not require data migration. If we must revert the entire version, we roll back to a build that still supports both modes and run it in legacy mode. In both cases, because the two modes persist sessions differently, users with active sessions may need to re-authenticate, but the service remains available.

Dual-Mode Testing

Every integration test must pass in both modes. The test infrastructure runs the full suite twice (controlled by environment variables), once in legacy mode, and once in separated mode. This continuous validation ensures both code paths remain functional throughout development.

Performance Validation: Load Test Results

Storing the sessions in a separate datastore is elegant in theory, but production performance is what really matters. Extensive load testing was conducted to validate the new architecture under realistic conditions. Fortunately, the results exceeded expectations!

Test Environment

For enterprise-scale validation (simulating some of our largest customers’ traffic patterns), a test environment was configured targeting 7,500 logins/second:

• The operational database was MongoDB Atlas M80 with 6 shards and 5 replicas across 3 regions.
• The sessions database was a dedicated MongoDB Atlas M80 cluster with 6 shards and 3 replicas.
• FlexID pods were scaled to 1,740 available cores.

Key Findings

The most striking result was the 141ms reduction in p95 latency (from 392ms to 251ms). This improvement directly impacts user experience as authentication flows complete noticeably faster, reducing the perception of “waiting” during login. The improvement stems from eliminating monolithic document updates. In legacy mode, every session touch required loading and writing the entire user document. With separation, session operations hit a dedicated, optimized store.

The separated architecture achieved something the legacy approach couldn’t: load distribution across specialized stores. The operational database CPU dropped from 60% to 48%, while the sessions database handled its load at just 37% CPU. This headroom is critical for production stability as the legacy approach was pushing the operational database toward its limits, leaving no margin for traffic spikes.

With the separated architecture, throughput achieved was 15,383 requests/second compared to 14,777 req/sec in legacy mode. A 4% throughput increase. More importantly, the system maintained this throughput with lower resource utilization, indicating significant additional headroom for growth.

Results Across Database Backends

In a different test, we also validated the architecture with different database combinations:

The Redis-backed session store showed the best results for the operational database reducing MongoDB CPU utilization by 40% (from 45% to 27%) while maintaining equivalent latency.

Production Implications

Cost efficiency: Lower CPU utilization means smaller database tiers can handle the same load.

Reliability: 20% CPU headroom provides buffer for traffic spikes.

Scale ceiling: Higher sustainable throughput delays the need for horizontal scaling.

User experience: 36% faster p95 means fewer users experience slow authentications.

The separated architecture positions the FlexID platform for independent scaling of each component and flexibility to evolve storage strategies over time. More importantly, the load test results validated that the architectural investment delivered measurable, production-relevant improvements (not just cleaner code), with 36% faster authentication for end users of our customers.

Architecting for Scale: Session Separation in Modern Authentication Platforms was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Engineering Team, Transmit Security

This is the first post in a series on face recognition technology. In this post, we’ll explore the fundamentals, from how face recognition evolved to how modern systems actually work. Throughout this series, we’ll build a complete understanding of face recognition from theory to real-world implementation.

Face recognition is no longer futuristic. It’s part of our daily lives. From unlocking smartphones to passing through airport security, it quietly powers some of the most important moments in modern life. But how does it actually work? Behind the scenes is a fascinating blend of computer vision and AI.

In this blog post, we’ll break down key concepts that help reveal the true power and complexity of these systems and why understanding them matters in the identity and fraud prevention world.

Understanding the Core of Face Recognition: Detection, Verification, and Identification

Face recognition technology isn’t one-size-fits-all. At its core, it’s designed to answer three fundamentally different questions, each requiring distinct approaches and trade-offs. To grasp its full potential, it’s essential to break down the core components:

1. Face Detection: Is there a face?

This is the foundational step in any face recognition system. Detection identifies whether a face is present in an image or video and pinpoints its location using a bounding box. At this stage, there’s no recognition or identity, just the confirmation that a face exists.

2. Face Verification (1:1): Are you who you say you are?

Verification is a one-to-one comparison between two faces, typically between a live image and a stored biometric. It’s like matching a key to a specific lock. This method powers secure experiences like unlocking your phone or authorizing a high-value transaction in a banking app.

3. Face Identification (1:N): Who is this?

This is where the system compares one face to many — scanning a database to find a match. Identification enables powerful use cases like fraud detection, public security, and seamless customer recognition, even in complex environments like retail or transportation hubs.

Real-World Applications: Where Face Recognition Delivers Value

Face recognition delivers tangible value across industries, and its impact continues to grow. But not all face recognition systems are created equal. The priorities behind their design shift depending on the context of use.

• Cooperative Systems: When the user opts in

In cooperative scenarios, the person being recognized is an active participant. These systems are designed for convenience and speed. Think of unlocking a phone with your face, accessing your laptop without typing a password, approving a payment in a banking app or standing at a biometric passport gate at airport border control. The engineering focus here is all about delivering a smooth user experience, protecting privacy, and ensuring security in milliseconds.

• Non-Cooperative Systems: When recognition happens passively

These use cases involve recognizing individuals who may not be aware or engaged with the system. This includes surveillance cameras in public spaces, security monitoring at large venues, and law enforcement initiatives like identifying suspects from crowd footage. In these environments, performance hinges on high accuracy, scalability, and the ability to search vast databases under challenging conditions such as poor lighting or partial occlusion.

Critical Challenges in Face Recognition

Before exploring the technical pipeline, let’s examine the real-world challenges these systems face. As face recognition advances, so do concerns around security, fairness, and reliability.

• Presentation attacks (Spoofing): Attackers attempt to trick the system using high-resolution photos, video replays, or even 3D masks. These attacks target the sensor level of the system.
• Defense: Liveness detection helps confirm a real person is in front of the camera. This can be done actively, with prompts like blinking or head movement, or passively, using texture analysis or depth sensing through 3D or infrared cameras.
• Adversarial attacks: These involve subtle changes, like digital noise or specially designed accessories such as adversarial glasses, that confuse the AI model without being obvious to the human eye.
• Defense: Robustness is improved through adversarial training, where models are exposed to these manipulations during training to learn how to resist them.
• Algorithmic Bias & Fairness: Face recognition systems often show higher error rates for different gender, ethnic or age groups. Models trained on unbalanced datasets tend to underperform on underrepresented groups, which can lead to serious consequences such as false rejections or even wrongful arrests.
• The answer is not just collecting more data but ensuring the training data is diverse and representative of the populations the system will serve.

The Face Recognition Pipeline: A Multi-Stage Journey

Step 1: Face Detection — Finding Faces Fast

Every face recognition journey starts with detection. The system must first locate all faces in an image or video.

• Classical Methods: Lightweight techniques like Viola-Jones and HOG still run well on low-power devices but struggle with angled faces and varied lighting.
• AI-Powered Methods: Deep learning models like MTCNN, RetinaFace and YOLO-Face deliver high accuracy and speed. They detect faces in real time and can even map facial landmarks for downstream tasks.

Step 2: Face Alignment — Getting Faces Camera-Ready

Once a face is detected, alignment ensures it’s properly positioned for accurate recognition. Real-world images vary in angle, lighting and scale, so alignment brings consistency.

The process involves:

1. Finding Facial Landmarks: Key points like the eyes, nose, and mouth are identified, often by the detection model itself (e.g., MTCNN).
2. Applying Geometric Transformation: The face is rotated, scaled and shifted so features land in a standard position, typically with eyes level and centered.

Step 3: Feature Representation — Creating the “Embedding”

This is often considered the “heart of the system.” A deep Convolutional Neural Network (CNN), trained on millions of faces, transforms the aligned image into a compact numerical vector called an “embedding.”

• Faces are turned into a compact vector of values, often 128 or 512 floating-point numbers, representing key facial features.
• The model learns to focus on distinguishing traits like facial geometry, while ignoring less reliable ones like lighting or hair color.
• Faces from the same person generate similar vectors, while faces from different people are far apart. Once this vector is created, the original image is no longer needed for matching.

Step 4: Matching and Decision — Comparing Embeddings

The final stage involves simple mathematics: comparing embeddings and making a decision.

Calculate Similarity: The distance between two embeddings is measured using a metric.

• Euclidean Distance (L2): Measures the straight-line distance. A smaller distance indicates more similarity.
• Cosine Similarity: Measures the angle between vectors. A value closer to 1 indicates more similarity and is generally more robust to lighting changes.

Compare to a Threshold: The calculated similarity score is compared to a predefined threshold. If it passes, it’s considered a match.

The choice of this threshold is a critical business and product decision that balances security against convenience:

• False Acceptance Rate (FAR): An imposter is incorrectly recognized. This is a security risk.
• False Rejection Rate (FRR): A valid user is incorrectly denied access. This impacts user experience.

Finding the right balance depends heavily on the specific use case and system requirements.

The Evolution of Face Representation: Key Milestones

Before diving into how modern systems work, let’s trace the key breakthroughs that got us here. Each milestone fundamentally changed how computers ‘learn’ to recognize faces. Let’s explore three significant milestones in the evolution of how faces are represented:

Classical Approach: Eigenfaces (1991)

• How it works: Uses Principal Component Analysis (PCA) to find the directions of greatest variance in a dataset of faces. These directions, when visualized as images, are the “eigenfaces.”
• Representation: Any face can be represented as a weighted sum of these eigenfaces, and the vector of weights becomes the feature vector.
• What stands out: The first truly automated, holistic method, proving the concept of a “face space.”
• Where it falls short: PCA is optimized for reconstruction, not discrimination. It finds commonalities rather than unique features, making it highly sensitive to lighting and pose variations.

Deep Learning Game-Changer: FaceNet & Triplet Loss (2015)

• How it works: Trains a Convolutional Neural Network (CNN) using a “Triplet Loss” function.
• Training with Triplets: The network is fed three images at a time: an ‘Anchor’ face, a ‘Positive’ image of the same person, and a ‘Negative’ image of a different person.
• Loss Function Goal: To ensure that the distance between the Anchor and Positive images, plus a certain margin, is less than the distance between the Anchor and Negative images (dist(Anchor, Positive) + margin < dist(Anchor, Negative)).
• What stands out: Directly optimizes the embedding space for discrimination, achieving record accuracy with compact embeddings.

ArcFace & Angular Margin (2018)

• How it works: Modifies the standard classification loss (softmax) to enforce a larger angular margin between different identities.
• Geometric interpretation: All face embeddings are projected onto a high-dimensional hypersphere, and similarity is measured by the angle between vectors. During training, a margin is added to the angle (cos(θ + m)), making the classification task harder.
• Result: This forces features from the same person into tighter clusters (intra-class compactness) and pushes clusters of different people further apart (inter-class discrepancy).
• What stands out: Achieves better performance and is generally simpler and more stable to train than Triplet Loss.

The Bigger Picture

Face recognition is undeniably powerful, but technical sophistication alone doesn’t guarantee success. Real-world effectiveness hinges on how well systems navigate fraud, edge cases, privacy, and fairness, often simultaneously. Each stage of the face recognition pipeline adds value, but also introduces risk if not designed thoughtfully.

That’s why it’s no longer enough to treat face recognition as a standalone tool. The real value comes from integrating it into a broader identity and fraud prevention strategy — one that is powered by AI, designed for scale, and capable of adapting in real time to both user behavior and emerging threats. As these technologies become more embedded in our daily lives, getting this balance right is critical for both security and trust.

In an upcoming post in this series, we’ll tackle one of the most critical challenges head-on. We’ll explore practical strategies for ensuring face recognition systems perform fairly across all demographics and work equitably for everyone.

Unveiling Face Recognition: From Pixels to Identity was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let me tell you about the time we almost lost our minds trying to keep track of our cloud infrastructure at Transmit Security.

It started innocently enough. We had one product, a couple of environments, some AWS accounts, and a bit of GCP thrown in. Fast forward five years, and we were drowning in a sea of IaC configurations spread across dozens of product repositories with no clear standards.

“Which module version are we using in production again?” became a daily question. “Why does the staging environment have a different VPC setup than prod?” was another favorite. And my personal nightmare: “Why does this service have different permissions across different environments?”

Sound familiar? I thought so.

After having tons of conflicts that made us create new Cloud resources in several environments and import them to our state, we knew something had to give. That’s when we embarked on building ARCH, what I now consider our secret weapon.
Our Automated Reusable Configuration Hierarchy system, a comprehensive Infrastructure as Code management system that has completely transformed how we handle our Infrastructure code.

As a fast growing company with lots of products and environments, working on all major clouds, we needed a better solution for managing our infrastructure and scaling fast. Our identity and security solutions were expanding rapidly across AWS, GCP, and Microsoft Azure, and the traditional approach to infrastructure management simply couldn’t keep pace with our growth trajectory. We needed something that would eliminate bottlenecks, standardize deployments, and give our teams the agility to innovate without being bogged down by infrastructure complexities.

The Infrastructure Management Challenge

If you’ve worked with infrastructure at scale, you know the pain points:

• ❌ Configuration drift between environments
• 🧠 Knowledge silos where only certain team members understand certain parts of the infrastructure
• 🔁 Inconsistent implementations of similar resources across different projects
• 🛠️ Tedious, error-prone manual processes for creating new environments or services

Before our current system, our per product repositories were filled with different standards, inconsistent naming conventions, and a significant amount of duplicated and unorganized code. Making a simple change across all environments often meant updating dozens of files, with the constant risk of missing something important.

Our Solution: A Hierarchical, Templated IaC System

After evaluating various approaches, we designed a system that leverages Terragrunt and Gruntwork’s boilerplate to generate and maintain HCL files.

Core Concepts

• Define infrastructure values in a hierarchical and composable YAML structure
• Use templates to generate the actual HCL configuration files
• Automate everything with GitHub Actions

The Building Blocks

Central Repository

• HCL Templates: Organized by module and version
• default.yaml & globals.yaml: Common configs and latest module versions
• Reusable workflow: A GitHub Action for rendering templates

Product-Specific Infrastructure Repository

• Values: YAML files structured by product → cloud (AWS/GCP/Azure) → environment (DEV/STG/PROD) → region → project (Region’s main shared project, A specific customer’s project, etc.)

Teams work in their own repos, with ARCH generating the right HCL structure behind the scenes.

Managing Terraform Module Compatibility with Version-Aware Templates

Terraform modules evolve over time — sometimes with breaking changes that require different inputs, outputs, or structural definitions. To handle this without cluttering our logic or forcing “if-else” chaos in templates, we introduced version-aware HCL templates.

Directory Layout Based on Module Major Version

We structure our templates directory based on the major version of the Terraform module. Each versioned folder contains a complete, valid HCL template that maps to the expected structure of that module version.

templates/
└── bigtable/
├── v0.x/
│ └── bigtable.hcl
└── v1.x/
  └── bigtable.hcl

ARCH picks the right template automatically based on the module version in your YAML.

How It Works

Our YAML spec (the desired state) includes the desired module version (e.g., v1.2.3). During the rendering phase, the GitHub Action:

1. Parses the module version.
2. Extracts the major version (e.g., v1.x from v1.2.3).
3. Selects the correct template path based on the major version.
4. Renders the appropriate HCL using the matching template and YAML input.

This approach allows us to support multiple versions of the same module in parallel — critical when different environments or products are on different upgrade cycles.

How It All Works Together

The magic happens when we combine these components. Here’s the flow:

1. 🧩 Engineers update YAML values at the correct hierarchy level
2. 🤖 GitHub Action merges configs (default → globals → product → cloud → environment → project → region)
3. 🏗️ HCL templates are rendered using the final merged configuration
4. 📤 Pull request with the generated HCL is created automatically

This approach means that each service’s HCL file contains everything needed for deployment, making the process streamlined and less prone to errors.

Why This Changed Everything

This system has fundamentally changed how we manage infrastructure, delivering several key advantages:

1. Simplified Configuration Management

Define common settings once, override when needed.

Take a look at this folders tree:

values/
├── default.yaml # Default Terraform module versions (Taken from the central repo)
├── globals.yaml # Organization-wide settings (Taken from the central repo)
├── product.yaml # Product-specific settings
├── <cloud>/
│ ├── cloud.yaml # Cloud-specific configurations
│ ├── <environment>/
│ │ ├── environment.yaml # Environment settings
│ │ ├── <project>/
│ │ │ ├── project.yaml # Project specific configuration
│ │ │ ├── regions/
│ │ │ │ ├── region.yaml # Region specific configuration
│ │ │ │ ├── <region>/
│ │ │ │ │ ├──values.yaml # Specific configuration

This hierarchy allows us to make broad changes at higher levels while maintaining the flexibility to customize at more specific levels.

2. Consistency Across All Environments

Our pre-built templates ensure that all services are defined consistently, regardless of which team or engineer set them up. This consistency extends across environments, regions, and cloud providers.

For instance, a PostgreSQL database in development will have exactly the configuration structure it needs and will be as close as it can to the one in production, with only the specific values differing based on environment needs.

3. Seamless Scaling

Want to provision a new environment or cloud? Just add a few YAML files.
The system takes care of the rest in minutes.

This scalability has been particularly valuable as we’ve expanded to new regions and added new products, allowing us to maintain a consistent infrastructure approach throughout our growth.

4. Reduced Error Rates

With automation handling the complex parts of infrastructure generation, we’ve seen a significant decrease in deployment errors. The system ensures that all dependencies are properly defined and that all required values are present before deployment.

5. Centralized place for common configuration

Need to whitelist new IPs of a vendor?
Add them once, and they propagate across all products.

6. Eliminating infrastructure drifts

To further reduce the risk of configuration drift and ensure our infrastructure remains up to date with our desired state, we implemented a periodic automation that executes the same infrastructure generation pipeline (generate-infrastructure action) for all relevant environments and opens a Pull Request only if there are changes.

📍Real Example

Let’s take a BigTable template for example, its template will look like that:

terraform {
  source = "<TERRAFORM_MODULES_URL>?version={{ .gcpBigtable.TFVersion }}"
}

locals {
  {{- range .gcpLocals }}
  {{ .name }} = "{{ .value }}"
  {{- end }}
}

include "root" {
  path = find_in_parent_folders()
  expose = true
}

inputs = {
  project_id = {{ .gcpBigtable.project_id }}
  cluster_name = "{{ .gcpBigtable.cluster_name }}"
  zones_list = {{ indent 2 .gcpBigtable.zones_list }}
  region = "{{ .gcpBigtable.region }}"
  {{- if hasKey .gcpBigtable "storage_type" }}
  storage_type = "{{ .gcpBigtable.storage_type }}"
  {{- end }}
  {{- if hasKey .gcpBigtable "autoscaling_parameters" }}
  autoscaling_parameters = {{ indent 2 .gcpBigtable.autoscaling_parameters }}
  {{- end }}
  {{- if hasKey .gcpBigtable "bigtable_members_list" }}
  bigtable_members_list = {{ indent 2 .gcpBigtable.bigtable_members_list }}
  {{- end }}
  {{- if hasKey .gcpBigtable "fixed_node_num" }}
  fixed_node_num = {{ .gcpBigtable.fixed_node_num }}
  {{- end }}
  {{- if hasKey .gcpBigtable "kms_rotation_period" }}
  kms_rotation_period = {{ .gcpBigtable.kms_rotation_period }}
  {{- end }}
}

When merging the yaml files in the hierarchy for product named “product”, in our “prod” environment “main” project in “gcp” cloud, located in “us-east1” region and getting file containing the following:

globalLocals:
  productName: "product"
  environmentName: "prod"
  projectName: "main"
  regionName: "us-east1"

gcpBigtable:
  TFVersion: 1.2.3
  project_id: dependency.project.outputs.projects.project_id
  cluster_name: "{{ .globalLocals.productName }}-{{ .globalLocals.environmentName }}-{{ .globalLocals.projectName }}-{{ .globalLocals.regionName }}"
  region: "{{ .globalLocals.regionName }}"
  zones_list: |
    [
      "{{ .globalLocals.regionName }}-a",
      "{{ .globalLocals.regionName }}-b",
      "{{ .globalLocals.regionName }}-c"
    ]
  storage_type: "SSD"
  autoscaling_parameters: |
    {
      min_nodes_number = 4
      max_nodes_number = 500
      cpu_target_percent = 35
      storage_target_percent = 8192
    }

The generated HCL file will look like this:

terraform {
  source = "<TERRAFORM_MODULES_URL>?version=1.2.3"
}

locals {
  product_name = "product"
  cloud_name = "gcp"
  env_name = "prod"
  region_name = "us-east1"
  project_name = "main"
}

include "root" {
  path = find_in_parent_folders()
  expose = true
}

inputs = {
  project_id = dependency.project.outputs.projects.project_id
  cluster_name = "product-prod-main-us-east1"
  zones_list = [
    "us-east1-a",
    "us-east1-b",
    "us-east1-c"
  ]
  region = "us-east1"
  storage_type = "SSD"
  autoscaling_parameters = {
  min_nodes_number = 4
  max_nodes_number = 500
  cpu_target_percent = 35
  storage_target_percent = 8192
  }
}

In all other environments that will use BigTable, the generated HCL will be built with the same standardization, that’s how we keep consistency while still preserving the flexibility of every environment to set its needed configuration.

✅ Real-World Impact

We tested ARCH by migrating an old environment and creating a new one.

• Initial migration? Took time — but taught us how to structure everything.
• Next environment? A single YAML file and we were done.

Results:

• We cut the time for setting up infrastructure for new projects by roughly 85%. What used to take a working day can take couple of minutes if we configure things correctly.
• Configuration drift between environments? Pretty much eliminated. When we find a discrepancy, we fix it at the appropriate level in the hierarchy once, and boom — it propagates everywhere it needs to.
• New team members are getting up to speed much faster. Previously, it would have taken a month just to understand our setup.
• Our SecOps is much happier now that security configurations are applied consistently across all environments.

Bonus Benefits

• Team mobility. Now we can move engineers between teams without a massive ramp-up period. Since the infrastructure follows the same patterns across all products, a developer who’s been working on Product A can quickly contribute to Product B without learning a whole new infrastructure approach. This has been a game-changer for resource allocation.
• Automated module upgrades. We set up a periodic job that updates module versions in our hierarchy, generates the new configurations, and creates PRs. This means we can quickly roll out security patches or new features across our entire infrastructure with minimal manual work.
• When a critical vulnerability was announced in one of our cloud providers recently, we were able to patch all affected resources across our entire organization in under a day. Before this system? That would have been weeks of work and plenty of missed instances.

Lessons Learned & Best Practices

Building this system wasn’t without challenges. Here are some key lessons we’ve learned:

The “Define Once” Philosophy

We quickly learned that the single most important practice is our “define once” philosophy. Every value belongs at exactly one level in the hierarchy — the highest level where it makes sense.

Need to change a default timeout that affects all products? Change it in globals.yaml.
Need to update something specific to AWS in all environments? That goes to cloud.yaml file.

This approach means that when we need to make a change that affects multiple services or environments, we make it in one place. Next month, we’ll need to update our logging configuration across all products. One change to globals.yaml, one PR, and it will automatically be propagated everywhere. The old way? Probably 10+ PRs, change priority and context of members in every team and days of work.

Self-Contained Definitions Save Sanity

No more jumping through files to debug a resource.
Everything is in one file.

Automate Everything (Seriously, Everything)

Let automation be your best team member.
Our GitHub Actions merge, render, validate, and open PRs — so we don’t have to.

Conclusion

I won’t pretend implementing this system was easy. We spent weeks designing the hierarchy, creating the initial templates, and building the automation. There were heated debates about where certain values belonged. And yes, there was pushback from teams comfortable with their existing approaches.

If you’re struggling with similar challenges, I’d encourage you to consider a hierarchical approach like ours. Start small, perhaps with just one product or environment, and gradually expand. Focus on building good templates and automation from the beginning. And remember that the goal isn’t just consistency for consistency’s sake — it’s about enabling your teams to move faster with confidence.

The cloud infrastructure world isn’t getting any simpler. But with the right systems in place, it doesn’t have to be chaotic.

________________________________________________________________
Feel free to drop questions in the comments if you’re tackling similar challenges.

ARCH: The NextGen IaC Generator was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR: At Transmit Security, we developed a centralized system for upgrading Kubernetes DevOps controllers across multiple products and environments. Our solution combines GitHub Actions, custom Helm charts, and automated processes to streamline upgrades, enhance security, and improve consistency. This article details our approach, its implementation, and its real-world impact on our operations.

_________________________________________________________________

In the fast-paced world of Kubernetes orchestration, staying current with controller versions is not just a best practice — it’s a necessity for maintaining robust, secure, and efficient infrastructure. Here at Transmit Security, where we manage a diverse portfolio of products across numerous Kubernetes clusters, this challenge is magnified by the scale and complexity of our operations. Today, we’re pulling back the curtain on an innovative approach we’ve developed to centralize and streamline the upgrade process for Kubernetes DevOps controllers across our multiple products and environments, a solution born from the unique demands of our robust infrastructure.

The Challenge: Taming the Multi-Controller Beast

Before we dive into our solution, let’s paint a picture of the challenges we faced:

1. Version Inconsistency: With multiple products and environments, keeping track of controller versions was like herding cats.
2. Manual Overhead: Upgrades often required manual intervention, eating into valuable developer time.
3. Lack of Standardization: Different teams had different upgrade processes, leading to potential errors and inconsistencies.
4. Scale Issues: As our infrastructure grew, so did the complexity of managing upgrades across the board.
5. Security Concerns: Delayed upgrades could potentially expose our systems to known vulnerabilities.

We needed a solution that would address these challenges while providing flexibility and control to our DevOps teams. Enter our centralized upgrade pipeline.

Our Solution: A Symphony of Automation and Control

We’ve architected a comprehensive solution that harmonizes GitHub Actions, custom Helm charts, and automated processes into a streamlined upgrade symphony. Let’s break down the key components and processes:

Key Components: The Building Blocks

Charts

Devops-stack-charts: This is our maestro chart, conducting the orchestra of external charts. It contains:

• External charts (such as cert-manager, cluster-autoscaler, etc.) defined in the Chart.yaml
• A template file for each controller in the templates directory
• Custom resources for each controller’s chart

This elegant structure empowers us to efficiently manage and customize resources for each controller while maintaining a centralized approach, significantly enhancing our operational efficiency.

Application-sets-chart: Think of this as our sheet music. It contains:

• A template for a MultiSource ApplicationSet resource
• Per-product values files with ApplicationSet configurations
• Global defaults and a hierarchy (product_name > cloud_name > environment_name)
• Version specifications for controllers
• Git repository URLs and branches for controller-specific values
• Sync options for the ApplicationSet

This chart is instrumental in our ability to manage configurations consistently across diverse products and environments, ensuring uniformity and reliability in our deployments.

GitHub Actions: The Automation Ensemble

renovate-scheduled.yaml: Our update scout. Running periodically, it:

• Checks for new controller versions
• Triggers the publish-chart action when updates are found

This proactive approach ensures that our system consistently benefits from the latest controller versions, enhancing security and performance.

publish-chart.yaml: The Packaging and Publishing Maestro. It:

• Packages each chart from external helm repositories (e.g., Bitnami, Jetstack)
• Incorporates relevant custom resources from devops-stack-charts/templates
• Publishes the resulting chart to our internal artifact repository

This process ensures that our customized charts are readily available for deployment, streamlining our infrastructure management.

generate-applicationset.yaml: Our template virtuoso. This action:

• Generates ApplicationSet templates based on user parameters
• Creates a new branch
• Opens a PR in the specific product’s repository (e.g., mind-infra, riskid-infra)

By ensuring that new configurations undergo review before implementation, it maintains the integrity of our infrastructure.
This action triggers a script called get_target_repo.py that contains the mapping of a product name <> repository name and helps us understand to which repository should we open the PR.

apply-appset.yaml: This action is our deployment conductor. It:

• Runs in per-product repositories
• Applies an ApplicationSet based on user-provided parameters

Its role is pivotal in the bootstrapping process, as the ApplicationSet takes over the reins for automatic syncing post-deployment.

├──README.md
├──.github
│ ├── scripts
│   └── get_target_repo.py
│ └── workflows
│ │  ├── apply-appset.yaml
│ │  ├── generate-applicationset.yaml
│ │  ├── publish-chart.yaml
│ │  └── renovate-scheduled.yaml
├── application-sets-chart
│ ├── Chart.yaml
│ ├── values.yaml
│ ├── productA.values.yaml
│ ├── productB.values.yaml
│ └── templates
│   └── applicationset.yaml
├── devops-stack-charts
│ │  ├── Chart.yaml
│ │  ├── templates
│ │  │ ├── controllerA.yaml
│ │  │ └── controllerB.yaml
│ └── values.yaml
└── renovate.json

The Upgrade Process: A Well-Orchestrated Performance

1. Renovate Scheduler checks for updates
2. If new versions found, trigger Publish Chart Action
3. Pull the changed Chart
4. Add Custom Resources
5. Package the updated Chart
6. Publish to the artifact repository
7. Create PRs in Dev Values
8. Generate ApplicationSet
9. Create PRs in Product Repository
10. DevOps Team Review
11. If approved, Deploy to Dev
12. Test in Dev
13. If successful, Update Other Environments
14. Generate New ApplicationSets
15. Deploy to Staging/Prod

Let’s walk through this process:

1. Weekly Update Check:
   Periodically, our Renovate scheduler GitHub Action runs its reconnaissance mission, scouting for controller updates. This regular check ensures we’re always aware of the latest versions available.
2. Automated Chart Publishing:
   When Renovate detects updates, it triggers our publish-chart action. This process efficiently retrieves all charts from their repositories, unarchives them, and integrates our custom resources for each controller. The charts are then repackaged with these enhancements and published to our artifact repository. This streamlined approach ensures our deployment pipeline always has access to up-to-date, customized charts, ready for immediate use. By automating this process, we maintain consistency, reduce errors, and significantly improve our deployment efficiency.
3. Automated PR Creation — The Proposal Stage:
   As soon as new controller versions hit the artifact repository, our system kicks into high gear:
   Pull Requests are automatically created in each per-product values file, specifically in the development section within the application-sets-chart in our devops-stack repository. The generate-applicationset action runs for each product’s development environment, preparing the ground for potential upgrades. PRs pop up in per-product repositories, suggesting changes to the ApplicationSets for development environments.
   This automated PR creation serves as a proposal system, allowing teams to review changes before implementation.
4. DevOps Team Review — The Human Touch:
   While automation is at the heart of our system, we believe in the importance of human oversight:
   DevOps teams receive Slack notifications about new versions, keeping them in the loop. Teams review the proposed changes, considering factors like compatibility and potential impact. They make the call on whether to implement the new version in the development environment. This step ensures that despite the automation, humans remain in control of what goes into their systems.
5. Controlled Rollout: From Dev to Production After a successful deployment in the development environment, DevOps teams can confidently roll out the new version across other environments:
   They modify their per-product values file in the application-sets-chart (located in the DevOps-Stack repository), updating the version for the desired environments. Using the generate-applicationset GitHub action, they trigger the generation of new PRs for each cluster that needs the update. This process allows for a gradual, controlled rollout, enabling teams to monitor the impact at each stage.

ArgoCD: One Instance Per Kubernetes Cluster

A key aspect of our architecture is running a dedicated ArgoCD instance for each Kubernetes cluster. This approach offers several advantages:

1. Isolation: Each cluster’s configuration and state are isolated, reducing the risk of cross-cluster issues.
2. Performance: Dedicated ArgoCD instances ensure optimal performance for each cluster, preventing resource contention.
3. Scalability: As we add new clusters, we can easily scale our ArgoCD deployment without impacting existing clusters.
4. Security: Cluster-specific ArgoCD instances enhance security by limiting the blast radius of potential breaches.
5. Customization: We can tailor ArgoCD configurations to the specific needs of each cluster.
6. Resilience: Issues with one ArgoCD instance don’t affect other clusters, improving overall system resilience.

This architecture aligns perfectly with our centralized controller upgrade strategy, allowing for granular control and monitoring of the upgrade process across our infrastructure.

Per-Product Repository Structure

To give you a clearer picture of how our solution is organized, let’s take a look at the structure of our per-product repositories:

product-infra/
├── devops-stack/
│ ├── appsets/
│ │ ├── aws/
│ │ │ ├── dev/
│ │ │ │ ├── cluster-name/
│ │ │ │ │ └── applicationSet.yaml
│ │ │ │ └── another-cluster/
│ │ │ │ └── applicationSet.yaml
│ │ │ ├── staging/
│ │ │ └── prod/
│ │ └── gcp/
│ │ │ ├── dev/
│ │ │ ├── staging/
│ │ │ └── prod/
│ └── apps/
│ ├── aws/
│ │ ├── values.yaml
│ │ ├── dev/
│ │ │ ├── values.yaml
│ │ │ ├── cluster-name/
│ │ │ │ └── values.yaml
│ │ │ └── another-cluster/
│ │ │ └── values.yaml
│ │ ├── staging/
│ │ └── prod/
│ └── gcp/
│ │ ├── dev/
│ │ ├── staging/
│ │ └── prod/
└── README.md

This structure allows us to:

1. Organize resources by cloud provider (aws, gcp)
2. Separate environments (dev, staging, prod)
3. Manage multiple clusters within each environment
4. Keep product-specific resources separate from the DevOps stack
5. Maintain consistency across products while allowing for product-specific customizations

The applicationSet.yaml files in each cluster directory are the key to our centralized upgrade process, as they define which controller versions and configurations to apply to each cluster.

Benefits: Why This Approach is a Game-Changer

1. Centralized Management:

• Single source of truth for controller versions and configurations
• Easier auditing and compliance management
• Reduced cognitive load on DevOps teams

2. Automated Updates:

• Significant reduction in manual intervention
• Faster response to critical updates and security patches
• Reduced human error in the update process

3. Consistency Across the Board:

• Uniform controller versions across products and environments
• Standardized upgrade process for all teams
• Easier troubleshooting due to version consistency

4. Controlled Rollout:

• DevOps teams retain oversight and control
• Ability to test updates in development before production rollout
• Flexible upgrade schedules per product and environment

5. Scalability:

• Easily adaptable to new products and controllers
• Process remains efficient regardless of the number of clusters or controllers
• Reduced overhead as infrastructure grows

6. Enhanced Security:

• Faster implementation of security patches
• Reduced window of vulnerability due to outdated controllers
• Improved compliance with security best practices

7. Time and Resource Efficiency:

• DevOps teams can focus on strategic tasks rather than manual upgrades
• Reduced downtime and smoother upgrades
• Cost savings from optimized use of human resources

Real-world Impact: A Case Study

To illustrate the power of this approach, let’s look at the following scenario:

When a critical security patch was released for one of our key controllers, our Renovate action picked it up. The publish-chart action immediately packaged and published the updated chart to the artifact repository. PRs were automatically created for our development environments across all products.

Our DevOps teams, alerted by Slack notifications, quickly reviewed and approved the changes. Within a day, the patch was tested and deployed across all development environments. Using the controlled rollout process, teams then systematically updated staging and production environments over the next maintenance time.

What once would have been a long, stress-filled scramble to manually update controllers across our entire infrastructure was reduced to a smooth process with minimal manual intervention.

Conclusion: Embracing the Future of Kubernetes Management

By centralizing our Kubernetes DevOps controller upgrade process, we’ve not only streamlined our operations but also set the stage for more advanced automation and optimization in our Kubernetes ecosystem. This approach has transformed a once-cumbersome process into a streamlined, efficient operation that enhances our ability to maintain a cutting-edge, secure infrastructure.

As the Kubernetes landscape continues to evolve, so too will our processes and tools. By building a flexible, automated foundation, we’re well-positioned to adapt to whatever challenges the future of container orchestration may bring.

I’m excited about the possibilities this centralized approach opens up and committed to continual refinement and innovation in our DevOps practices.

_________________________________________________________________

I’d love to hear from you! Share your experiences, questions, or insights in the comments below and feel free to reach me here or via my LinkedIn.

From Chaos to Harmony: A Deep Dive into Centralizing Kubernetes Controller Upgrades was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

The secret tool research

Once we’ve deployed our ArgoCD servers in IaC way, in order to start using GitOps approach in our Kubernetes products with multiple environments (development, staging and production) it was the right time to handle the Secrets management challenge.

The legacy deployments in our products used the CI tool (Jenkins) with Jenkins credentials in order to push secrets to the helm values files or just manual creation (bad idea, we know that 🙂), so in order to create our Kubernetes charts on-top our new ArgoCD servers, we came to an understanding that we had to change our secrets management.

First step in solving the challenge would be — defining the problem and the requirements:

• Easy integration with ArgoCD
• Support HashiCorp Vault
• Working with all Kubernetes resources (some of our application charts includes secrets inside helpers and config maps)
• Integration with Helm

After a market research we’ve conducted, based on our requirements, we came to a conclusion that our choice is between Argo CD Vault Plugin and External Secrets Operator.

Because of ArgoCD Vault plugin being part of the ArgoCD ecosystem, we’ve chosen to begin our PoC with it.

How the Argo CD Vault Plugin being installed

Using Terraform value templating we’ve demonstrated in part 1, we can add vault plugin built-in integration, first creating a secret with vault authentication credentials with chosen authentication method, you can find the example here.

Adding it to the ArgoCD configuration template file using the following configuration.

How the Argo CD Vault Plugin Works

The Argo CD Vault Plugin works by replacing the pattern of <placeholder> in any kubernetes YAML or JSON manifests with the secrets values taken from Vault.

An annotation can be used to specify exactly the path that the plugin should look the Vault values. The annotation needs to be in the following format:

kind: Secret
apiVersion: v1
metadata:
   name: secrect-with-annotation
   annotations:     
      avp.kubernetes.io/path: "path/to/password"
type: Opaque
stringData:  
  password: <password>

Another way to specify the Vault secret path is using inline-path placeholders. This way allows us to specify path and key for multiple distinct secrets into the same YAML:

kind: Secret
apiVersion: v1
metadata:
  name: secrect-without-annotation
type: Opaque
stringData:
  username: <path:some/path/to/usernames#secret-key>
  password: <path:some/path/to/passwords#secret-key>

Once the plugin has finished replacing the placeholders with the secrets, ArgoCD applies the updated manifest to the Kubernetes cluster.

Prerequisites

In this tutorial we wont cover how to install Vault server, but you can find here the Vault installation documentation.

Deploy an ArgoCD application to development environment using the ArgoCD Vault Plugin with helm values file

First thing, we need to enable a Vault kv-v2 store:

$ vault secrets enable -version=2 argocd

Then, we need to create the kv secrets in vault for development environment.

Development secret:

$ vault kv put argocd/development username=my_dev_user password=my_dev_pass

===== Secret Path =====
argocd/data/development

======= Metadata =======
Key                Value
---                -----
created_time       2022-10-31T20:46:20.138968925Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Global secret:

$ vault kv put argocd/global globalToken=Mytoken

=== Secret Path ===
argocd/data/global

======= Metadata =======
Key                Value
---                -----
created_time       2022-10-31T21:03:07.745292476Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Now, let’s go back to our example of the helm chart application, which containing generic nginx with two templated secret manifests, env_secret.yaml and inline_path_secret.yaml. You can find this chart example on GitHub repository under the helm directory.

It’s worth paying attention to the templates secrets manifests, that represent two different ways of plugin usage.

annotation usage (env_secret.yaml):

apiVersion: v1
kind: Secret
metadata:
  name: my-secret-example
  annotations:
    avp.kubernetes.io/path: "argocd/data/{{ .Values.env }}"
type: Opaque
stringData:
  password: <password>

inline-path usage (inline_path_secret.yaml):

kind: Secret
apiVersion: v1
metadata:
  name: secrect-without-annotation
type: Opaque
stringData:
  username: <path:argocd/data/{{ .Values.env }}#username>
  token: <path:argocd/data/global#globalToken>

Finally, we need to create the ArgoCD application for development environment.

Can be saved as argo_app_dev.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-dev-app
  namespace: argocd # The namespace must match the namespace of your Argo CD instance - typically this is argocd.

finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: default # application namespace
    server: https://kubernetes.default.svc
  project: default # argo project
  source:
    path: helm # Path to appsPath with the template name
    repoURL: git@github.com:Transmit-Security-Engineering/argocd-with-secrets-mgmt.git # our repo with the chart example
    targetRevision: main
    plugin:
      name: argocd-vault-plugin-helm-with-args
      env:
        - name: HELM_ARGS
          value: -f dev.values.yaml # here we configured the values file of our environment
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
---

Create the ArgoCD application with kubectl:

$ kubectl create -f argo_app_dev.yaml

Time to check if we can see the application in the ArgoCD UI:

We can validate our secrets values using the following commands:

$ kubectl get secret my-secret-example -o jsonpath="{.data.password}" | base64 --decode
my_dev_pass%

$ kubectl get secret secrect-without-annotation -o jsonpath="{.data.token}" | base64 --decode
Mytoken%

$ kubectl get secret secrect-without-annotation -o jsonpath="{.data.username}" | base64 --decode
my_dev_user%

Changing the Vault Secret

What about the support of changing/rotating the secrets?

Time to figure out how it works!

First, update our development password secret in vault:

vault kv put argocd/development password=my_new_dev_pass

===== Secret Path =====
argocd/data/development

======= Metadata =======
Key                Value
---                -----
created_time       2022-11-01T00:20:08.127676478Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

Performing a manual hard refresh to the application via the ArgoCD UI, will make a dry run of the plugin:

Then, the ArgoCD Vault Plugin will update the secret from Vault. ArgoCD will perform a sync to the application and will update the Kubernetes secret.

And we’re good to go:

$ kubectl get secret my-secret-example -o jsonpath="{.data.password}" | base64 --decode
my_new_dev_pass%

So it’s time to make a decision.

In the beginning, the implementation of the ArgoCD Vault Plugin works great for us. The plugin support with all Kubernetes resources was the best feet for our helm application structure. In addition, the plugin did not require an Operator or Custom Resource to manage and maintain.

But then, we ran into some issues…

The problem started when some of our developers teams requested an auto sync secrets support.

Unfortunately, sync secrets in the plugin working in manual way or can be implement with custom webhook/CronJob (auto refresh of secrets probably won’t be included in the next Argo CD Vault Plugin versions —reference to the GitHub issue).

The sync disadvantage and the fact that the plugin can be configured to work with only one secret manager at time, brought us to start testing the second option, the External Secrets Operator.

In this post, you have learned how to deploy an helm chart with values file on ArgoCD server, taking the secrets from Vault and using the Argo CD Vault Plugin as our secret management tool.

On our next blog, we would like to focus on External Secrets Operator with ArgoCD integration — so make sure you read the third part!

Building ArgoCD Ecosystem With Secret Management — The GitOps Way (Part II) was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Digital transformation is on, full power!, and with that more organizations turn to either leveraging SaaS solutions, or migrating their own capabilities to a SaaS model.

What differentiates an OK SaaS provider from a great one? We refer to those as the OAS of SaaS, the superpowers of a great SaaS provider. When you have it — you are Superman. When you don’t, you are Clark Kent holding kryptonite…

So, what is OAS? OAS stands for Observability, Security and Availability. The three pillars for achieving a great SaaS operation with common denominator — in order to achieve excellence in any of those you would need to implement a layered approach solution.

On this blog we will touch on what each of those mean, whereas in following posts we will share in more detail how we implement each of those in Transmit Security.

Observability

Every organization, and especially a SaaS one, would like to be in a position of making decisions based on data. This will work well, provided:

1. You have good quality data
2. Your data is well organized in a way that will allow you to benefit knowledge and insights

Your Observability stack is a critical part of your immune system — much like in Biology, this immune system needs to be able to protect you by blocking / preventing, and alert you and notify you before things go really bad, so you will have time to react and implement the right medicine.

In a nutshell the Observability stack will consist of 3 types of systems:

• Data gathering technologies — collecting and storing metrics, logs etc.
• Presentation technologies — allowing you to arrange the data in a human easy to understand dashboards
• Alerting technologies — enabling you to get push alerts based on predefined thresholds and scenarios

Availability

If your service isn’t available, it is not generating any value. In a global world that operates 24x7 there is no room for a service that is not meeting the same level of availability.

The SaaS availability is driven by many underlying components, and eventually will be determined by the weakest link in the chain. To name a few: Compute power, Network infrastructure, Code resilience etc.

To assure you meet the desired availability, you would need to implement a layered approach:

First introducing Redundancy of your components (clustering your compute resources, leveraging more than one provider, traffic multi path, geographic spread)

Second in line is Recovery, protecting your service in case of failure such as data corruption, where keeping multiple synced copies of data will not save the day

Third layer is Testing. The redundancy and recovery operations implemented must be tested (and monitored) on an on-going basis — or you will find that they failed at the worst time ever. The most common examples for glorious failures, is a backup process with no restore test, or having a DR site which never received traffic and was left behind.

When you are skipping the third layer of Testing you are losing twice — on one hand you pay for overhead of having those in place, on the other hand, when you actually need it — it is useless.

Security

As part of the increasing digital transformation, we see more audiences acquiring and using online services. As a result we have more identities available online, and more fraud opportunities. And Indeed we see the increase in account take over (ATO) attacks.

Providing a service is a big responsibility. Protecting user information is even bigger, so one should not take it lightly, especially in light of the increasing threats.

To achieve your SaaS Security, as in any of the other OAS superpowers, you would need to implement a layered approach, starting with the first line of code you run.

Code security started with secured design, going through implementation, for example code scanning and vulnerability detection as part of the development process. Whether you choose to follow OWASP or other guidelines, the important part is not to ignore the findings and when needed prioritize fixes.

Second in line is the infrastructure security. Same here, regardless of the CSPM tool you are using — don’t underestimate the findings.

Third layer is the perimeter protection you should consider. Many different technologies should be considered here, to name a few: WAF (which should also provide you with DDoS protection), VPN (Keep inside what does not need to be outside…)

Now all you have to do is to go and build your OAS. Luckily, unlike Superman, it is not a trade acquired by birth, but capabilities you can build and develop, like a muscle.

It will require hard work and dedication — but this is also part of the fun in doing it. In coming posts we will share how we build our OAS SaaS muscles and powers — a never ending journey of continuance development and improvement.

OAS — The SaaS Provider Superpowers was originally published in Transmit Security Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.