Data Quality Beyond Tools: Leveraging DMAIC in Data Lakes
Data quality remains one of the greatest challenges in the modern data ecosystem. While the market offers an endless array of tools and libraries, quality issues continue to be a recurring pain point for most organizations.
The conversation often focuses heavily on tools and far too little on methodology. Without a well-defined framework, teams find themselves in a reactive cycle — patching incidents, creating ad-hoc tests, and accumulating technical complexity without ever addressing the root cause.
In this context, DMAIC emerges as a robust alternative. A structured five-stage methodology — Define, Measure, Analyze, Improve, and Control — DMAIC originated in Six Sigma to solve complex problems by focusing on the reduction of process variability.
The Context
At PagueVeloz, a fintech operating as the payment engine for Serasa Experian, we faced the challenge of migrating our Data Lake from Azure to AWS using Unity Catalog.
Despite modern tooling and unit testing, we struggled with recurring inconsistencies. In the financial sector, the stakes are high: repeated quality issues undermine user trust and pose direct business risks. It became clear that we needed more than just code; we needed a process.
D — Define
The first step was defining what “quality” meant for us. Instead of trying to “boil the ocean,” we applied the Pareto Principle and selected three critical domains: Customers, Card Operations, and Financial Records.
While we manage hundreds of tables from various sources, we decided not to dissipate our energy on generic tests across the entire Lake. Instead, we focused where the impact is vital — the company’s core. For these tables, we defined rigorous objectives and invested significantly more effort, while the remaining tables followed basic integrity checks.
This approach aligns with the efficiency mindset advocated by Elon Musk when describing SpaceX’s engineering processes:
“The most common error of a smart engineer is to optimize a thing that should not exist.”
Following this logic, we must remain skeptical and constantly ask: Does it make sense to create this test right now? Is this requirement actually intelligent? Excessive monitoring of irrelevant data only generates noise and alert fatigue. With DMAIC and a Lean culture of continuous improvement, the focus shifts: you identify and refine only the tests and monitoring that truly need to exist.
M — Measure
“If you can’t measure it, you can’t manage it.” — Peter Drucker
Without clear metrics and a distinct understanding of what constitutes a problem, improving data quality is impossible. In this stage, we quantified our current performance. In addition to latency (replication time), we introduced DPMO (Defects per Million Opportunities):
DPMO = (Defects / (Units × Opportunities per unit)) × 1,000,000
In a data context, a “defect” might be a divergent record or an incorrect value. Since our ingestion is near real-time, we used this stage for observation rather than immediate correction. We collected metrics over a prolonged period to understand the natural variation of the process before intervening.
A — Analyze
Analysis is the search for patterns. By cross-referencing latency with financial divergence, we identified that our quality level was near 4.5 Sigma. While acceptable to many, it was below our target.
Get FABIANO TEICHMANN’s stories in your inbox
Join Medium for free to get updates from this writer.
The analysis revealed a blind spot in the Silver layer upsert: the versioning field generated by the application failed during manual correction scenarios. Additionally, we found that transactions with the same ID processed within milliseconds of each other caused non-deterministic behavior during ordering.
I — Improve
Once the root cause was identified, we implemented structural improvements:
- Implemented timestamps captured by the DMS (CDC), making ordering far more reliable.
- Adjusted ordering criteria using window functions to ensure total determinism in the upsert process.
- Reprocessed the Bronze layer to eliminate traces of past divergences.
These improvements happened at distinct moments; as we measured and analyzed, we continuously uncovered new opportunities for refinement.
C — Control
Com certeza. Ter evidências claras e documentadas é fundamental para demonstrar maturidade técnica (especialmente em metodologias como Six Sigma/Lean).
Aqui está a tradução dos trechos que você forneceu, adaptada para um tom profissional e técnico de Data Engineering/Quality:
English Version (for LinkedIn or Medium)
C — Control The “Control” phase is where most data projects fail. Implementing an improvement without ensuring its sustainability is like “treading water.”
Our control process is now daily and automated. Every night, an audit job fetches the monetary value and record count from the original source, comparing them against the Data Lake.
We conducted long-term monitoring to ensure quality would be maintained over time. We identified some issues with the financial records table, but we were able to pinpoint the root cause and implement permanent improvements.
The Key Differentiator: Distinguishing Noise from Real Problems
- Monetary Divergence: We report the exact impact in local currency (BRL). Stating that “0.005% of data diverges” creates uncertainty; stating that “the financial divergence is $0.00” builds trust.
- DPMO (Defects Per Million Opportunities): This allows us to track the evolution of integrity. Since we work with streaming, small latency fluctuations are expected “noise.” DPMO helps us isolate this noise and trigger alerts only when there is a structural failure in quality.
Conclusion
Data quality is not a destination; it is a continuous process. By applying DMAIC, we elevated our core tables from 4.5 Sigma to the 6 Sigma range — an exceptional milestone for near real-time pipelines.
However, the greatest challenge of this methodology is not technical complexity, but culture. We often hear that “quality is a priority,” but in practice, technical teams often face resistance from management to prioritize structural improvements over new business demands.
The result of this neglect is a vicious cycle of “War Rooms”: the problem scales, the business complains, everything stops to apply “band-aid” solutions (which often create new problems), and promises of future monitoring are made — only to be forgotten weeks later.
The Lean Six Sigma approach breaks this cycle. It shifts the mindset to focus on the essential, investing energy strategically. Our greatest indicator of success wasn’t just a statistical number: it was the fact that, in two years, we never had to create a single War Room. While users still report issues, we use them as input for the improvement cycle, without the desperation of “dropping everything” for a single error.
To Tech Leads, Engineers, and Architects, I offer a suggestion: measure how much time your team spends “extinguishing fires.” Calculate the financial impact of incidents and, most importantly, the emotional toll on your team. How many great talents does the market lose to burnout caused by technical instability?
The effort to implement a DMAIC vision versus implementing dozens of aimless tests is virtually the same — but the results are worlds apart. Tools are merely the means; methodology is what ensures the data at the end of the pipe is a truly strategic and reliable asset for the business.
Would you like me to suggest a few different title options for your Medium post to help with SEO and engagement?
