Beyond Basic Validation: Building a Configurable Data Quality Framework with PySpark and AWS Glue

Building a reusable validation engine with YAML rules, automated quarantine, and queryable quality results on AWS.

When a Successful ETL Job Isn’t Enough

A data pipeline can execute successfully while still producing unreliable data.

Consider financial transactions containing missing identifiers, negative amounts, unsupported currencies, or duplicate transaction IDs. The ETL job might complete without errors, but the resulting data could compromise downstream analytics.

To explore this problem, I built and deployed a configurable data quality framework using PySpark and AWS Glue 6.0.

The objective was straightforward: validate transactions using reusable rules, separate invalid records without losing them, and generate measurable data quality results.

1. Architecture: From Raw Data to Validated Outputs

The framework uses Amazon S3 for storage, AWS Glue for processing, the Glue Data Catalog for table metadata, and Amazon Athena for SQL verification.

Figure 1: Architecture of the configurable data quality framework on AWS

The pipeline follows three stages:

1. Input: Read financial transactions from S3 and load YAML validation rules.
2. Processing: Execute a custom PySpark rule engine inside an AWS Glue job.
3. Output: Write valid transactions to curated Parquet, invalid transactions to quarantine Parquet, and quality metrics to JSON.

The curated and quarantine datasets are registered in the Glue Data Catalog, allowing Athena to query both.

2. Making Validation Configurable with YAML

Instead of embedding every validation requirement directly into the ETL job, the framework defines rules in a YAML configuration stored in S3.

For example, a dataset might require positive transaction amounts, supported currencies, valid dates, and unique transaction identifiers.

Figure 2: YAML configuration defining reusable data quality rules for financial transactions, including required fields, positive amounts, allowed values, date validation, and duplicate detection.

The complete YAML configuration and rule-engine implementation are available in the GitHub repository.

This separation allows validation requirements to change without rewriting the main Glue orchestration logic, provided the rule engine already supports the relevant rule type.

It also makes the framework easier to reuse across datasets.

3. Inside the PySpark Validation Engine

The custom rule engine evaluates configured conditions and records failed validations in a failure_reasons array.

For duplicate detection, the implementation uses Spark window functions to identify transaction IDs appearing more than once.

from pyspark.sql import functions as F
from pyspark.sql.window import Window

window = Window.partitionBy("transaction_id")
duplicate_flag = (
    F.count(F.lit(1)).over(window) > 1
)

Unlike dropDuplicates(), which retains one occurrence, this approach can identify and quarantine every record sharing a duplicated transaction ID.

That distinction matters in financial datasets: automatically choosing one duplicate as valid could conceal an unresolved transaction inconsistency.

After validation, records are separated according to whether their failure array is empty.

• Curated: Transactions that pass every configured rule.
• Quarantine: Transactions that fail at least one rule, retaining the reasons for rejection.

Rather than silently dropping problematic records, the framework keeps them available for investigation.

4. Deploying the Framework with AWS Glue

I deployed the implementation as an AWS Glue 6.0 PySpark job.

Get Utsavgautam’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The job reads the transaction CSV and YAML configuration from S3, imports the packaged validation engine, executes the rules, and writes the results to three S3 destinations:

• curated/ — Valid transactions in Parquet format
• quarantine/ — Invalid transactions with failure reasons
• metrics/ — JSON quality report

The reusable rule engine was packaged as a Python ZIP dependency and supplied to the Glue job.

Figure 3: Successful AWS Glue 6.0 execution

The test execution completed successfully in approximately 1 minute 30 seconds. This was a small functional demonstration, not a performance benchmark.

5. What the Actual Results Showed

To test the framework, I used 15 synthetic financial transactions containing deliberately introduced data quality issues.

The execution generated the following results:

The JSON report recorded failures involving missing transaction IDs, invalid amounts, unsupported currencies, invalid dates, unsupported statuses, and duplicate identifiers.

Two records were flagged for duplicate transaction IDs.

Figure 4: Generated JSON report showing validation metrics

The dataset deliberately included invalid transactions to test each validation rule. Therefore, the 53.33% pass rate demonstrates the framework’s ability to identify and isolate problematic records, rather than reflecting real-world data quality.

I then queried both outputs through Amazon Athena.

The curated table returned eight accepted transactions, while the quarantine table returned seven rejected transactions alongside their failure_reasons.

Figure 5: Amazon Athena query results showing eight validated transactions in the curated dataset

Figure 6: Athena results showing quarantined transactions and validation failures

This provided an independent check that the pipeline had correctly separated accepted and rejected records.

6. Technical Lessons and Production Considerations

Building the framework highlighted several important engineering decisions.

Duplicate handling: Rejecting every duplicate is a conservative policy. Production systems may instead require business-defined reconciliation logic.

Spark performance: Calculating individual rule metrics with repeated .count() actions can become expensive at scale. More efficient aggregation would be worth exploring for larger datasets.

Schema consistency: During Athena testing, a mismatch between the Data Catalog schema and the Parquet column types caused a query failure. Correcting the table schema resolved the issue, reinforcing the importance of consistent data types.

Operational readiness: A production deployment would need additional automated testing, monitoring, access controls, idempotency, and recovery mechanisms.

Conclusion

A successful ETL execution confirms that the processing job completed. It does not guarantee that the data meets business expectations.

This project demonstrates how configurable validation rules, PySpark processing, quarantine storage, and AWS analytics services can work together to make data quality measurable and traceable.

The key lesson: Don’t just monitor whether your pipeline succeeded. Measure whether the data can be trusted.

Source code and sample dataset: AWS Financial Data Pipeline — Data Quality Framework

Technologies: AWS Glue 6.0, PySpark, Amazon S3, AWS Glue Data Catalog, Amazon Athena, Python, and YAML.