Designing a Data Platform for 20+ Products: Standardization Without Losing Flexibility

Why the goal of a reusable data platform shouldn’t be making every pipeline identical.

Building the first data pipeline is usually straightforward.

Building the twentieth one is where architecture starts to matter.

When multiple products feed the same data platform, there is a natural temptation to create a separate pipeline for each product.

It works.

Until it doesn’t.

Soon you have:

Product A → Pipeline A
Product B → Pipeline B
Product C → Pipeline C
...
Product T → Pipeline T

Each pipeline has its own code.

Each has its own configuration.

Each has its own failure modes.

And a change to common logic suddenly means changing many implementations.

That was one of the problems we wanted to solve while modernizing a multi-product data platform.

The commonality was hiding in plain sight

Although the products were different, many of their pipelines followed the same fundamental pattern.

For example:

Source
  ↓
Extraction
  ↓
Bronze
  ↓
Transformation
  ↓
Silver

The source system might change.

The entity might change.

The mapping might change.

But the overall mechanics were often remarkably similar.

That suggested a better approach:

Build the pipeline once. Configure the differences.

Moving toward a configuration-driven framework

Instead of writing product-specific extraction logic repeatedly, we moved toward a shared Lambda-based ingestion framework.

The framework could use configuration to determine things such as:

• Source system
• Database type
• Product-specific mapping
• Extraction behavior
• Target entity
• Processing requirements
• This changed onboarding.

Adding another product could become primarily a configuration exercise instead of a completely new engineering project.

That’s an enormous difference when you’re operating at the platform level.

But standardization has a trap

There is a problem with saying:

“Every pipeline must use exactly the same pattern.”

Real systems don’t work that way.

Eventually, you encounter a product whose data model is different.

Or its identity information isn’t sufficient.

Or it has a dependency on another product.

Or its source system behaves differently.

Trying to force every exception into the standard template can make the platform less maintainable.

So the goal shouldn’t be:

100% standardization.

It should be:

Standardize the common path and make exceptions explicit.

A real example: identity dependency

One of the interesting cases involved a product whose customer records didn’t contain enough information to reliably determine the canonical customer identity.

The required identity mapping existed in another trusted product dataset.

So the pipeline followed the common ingestion pattern initially:

Source
  ↓
Lambda
  ↓
Bronze

But during the transformation stage, it needed an additional cross-product lookup.

Conceptually:

Product A customer
       ↓
Identity matching
       ↓
Trusted customer dataset
       ↓
Canonical identity

Records that couldn’t be confidently matched were not silently discarded.

Get Priya Jangid’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

They were surfaced for review.

That distinction is important.

A platform becomes dangerous when it turns uncertainty into false certainty.

The importance of an escape hatch

A reusable framework needs extension points.

Otherwise, every exception becomes a battle between:

“Break the framework”

and

“Break the business requirement.”

Neither is a good choice.

A better design is:

                    ┌─ Standard path
Input → Framework ──┤
                    └─ Controlled exception

The standard path should handle the majority of cases.

Exceptions should be:

• Explicit
• Documented
• Testable
• Observable
• Justified

That way, the exception becomes part of the architecture rather than hidden logic inside someone’s code.

The hidden cost of duplicated pipelines

Duplicated pipelines don’t just increase code.

They increase:

Maintenance cost

A bug fix needs to be replicated.

Testing cost

Every implementation needs validation.

Operational complexity

Engineers need to understand multiple versions of the same concept.

Onboarding cost

New engineers need to learn multiple patterns.

Risk

One pipeline may receive a fix while another quietly remains outdated.

A shared framework doesn’t eliminate these problems completely, but it can significantly reduce them.

But don’t build an abstraction too early

There is another lesson here.

Not every two pipelines need to become one framework.

Abstraction itself has a cost.

If two pipelines only look similar superficially but have fundamentally different requirements, forcing them into one abstraction can produce a complicated framework that nobody understands.

So before building a generalized pipeline framework, ask:

What is genuinely common?

Not:

What can I technically make common?

That’s an important distinction.

The principle I now follow

After working on multi-product data platforms, I think about standardization this way:

Make the common path boring.

The standard case should be predictable.

The configuration should be understandable.

The operational behavior should be consistent.

And when something genuinely doesn’t fit, the platform should provide a clean way to handle it.

The goal isn’t to make every pipeline identical.

The goal is to make the platform easy to extend without making it fragile.

And when the 21st product arrives, you should be asking:

“What configuration does it need?”

not:

“How many files do we need to copy from the last pipeline?”