A Secure Data Lake: AWS S3 and Snowflake
Part 1: Using AWS S3 as a Raw Data Zone in a Secure Data Lake Architecture
Introduction
Throughout history information has always affected our decision making, and because of that it holds great importance. The more information you have about a subject, the more of an informed decision you can make. In today’s world, we’re collecting information from everywhere, historical records and consumer IoT devices like smartphones, smart assistants, wearables, and even our cars. We inventory our groceries, we track our fitness goals, advertisers reach us with preferred products, and businesses define their quarterly strategies.
Having these deep and profound effects on us makes the creation of information a very important process. The need to house and secure this vast amount of information has become increasingly difficult and proving to have greater importance as Machine Learning and AI takes us into the next phase of our technological evolution.
Information and Data
Breaking it down, information is organized and processed facts and figures, and with given context, it helps you derive an understanding about a subject. Those facts and figures are the raw data. With the massive amounts of data that comes in from all the different sources and in all types of forms, a problem of storage and protection becomes evident. If we rewind just a few years ago, Relational Databases were the primary backbone behind most systems, and a combining and curation of that data in Data Warehouses were able to help us with our reporting and dashboards. Today, data structures don’t always allow us to use traditional databases and warehouses which adds to our data storage problem.
Data in the Cloud
Enter the cloud. While these traditional data systems are still used today for those same functions, their product casing and how they get their data looks different. They mostly live in the cloud, and because of cloud providers' massive infrastructure and ability to offer expandability, cloud repositories have emerged as the primary raw data storage source.
While that creates a way to store data, the problem of protection and access management still exists.
Welcome Data Lakes. You can think of a data lake as a central place of gathering all the raw cloud data storage, and creating a layer of security and management capabilities around them. Once the data is there, you can have back-end processes that run to change the form of that data for other downstream processes and submit that data to groups or departments to aid in the effort of reporting and visualization. This information can now affect the accuracy of everything, shopping, bill pay, even our fitness goals. Now for businesses, it can shift quarterly strategy, and sometimes even the workforce.
AWS S3 and Snowflake
A great combination of tools to store and work with big data are AWS S3 and Snowflake. I’ve personally come to love how they seamlessly fit together to give you an easy experience of creating a data lake backbone for your business or organization.
Amazon S3 is a great choice for cloud data storage.
- First, its API provides an easy way to work with inputting data into the service.
- Second, its durability promise provides assurance of data availability, and all while the service is giving you controls to manage and protect the data at its lowest level.
Now to work with, analyze, and provide that data downstream, enter Snowflake, a powerful cloud-based data platform. With Snowflake’s Organizations feature, you can centrally manage multiple Snowflake accounts across business units, environments, and regions. This deployment model supports a hub-and-spoke architecture, enabling consistent governance, data sharing, and cost controls.
Get Christopher B. London’s stories in your inbox
Join Medium for free to get updates from this writer.
In this three-part series, we’ll talk about how to build a secure data lake using Amazon S3 as a raw data store, Snowflake as the compute and query engine for processing, and AWS VPC Endpoints to enhance security and tie it all together. In Part 1, we’ll start with the role of S3 in a secure, scalable data lake.
Why Use AWS S3 as a Raw Data Zone?
Amazon S3 is durable, as in its well-architected framework it defines this system characteristic as:
A system that is durable is able to perform its responsibilities over time, even when unexpected events may occur. For example, a durable storage system will reliably store data without data loss.
AWS S3 is designed for 99.999999999% (11 nines) of durability.
This makes it ideal for a raw data zone, the initial spot for data ingested from applications, logs, IoT devices, third-party APIs, or legacy systems.
Highlights:
- Cost-Effective: S3 offers multiple storage classes (Standard, Infrequent Access, Glacier) to optimize for access patterns and cost.
- Scalable: Ingest data of any volume, velocity, or variety without managing infrastructure.
- Secure: Fine-grained access control with IAM, bucket policies, and encryption options.
Designing a Raw Data Zone in S3
The phrase data zones represents the different stages that data goes through in order to become curated and usable. A raw data zone represents the beginning stage where data is first ingested, and there are minimal to no changes. An example of a typical raw data change, is a change on ingestion, where maybe only masking PII for compliance or agreements.
A well-organized raw zone sets the foundation for downstream analytics. Here are a few considerations when designing:
- Naming Convention- Tagging and Metadata
Tag your buckets using clear, consistent prefixes. Allow IaC to name resources, and include in code a parameter for AWS SSM or Hashicorp Vault pointing to the resource name and uri:
example uri: s3://company-datalake/raw/<source_system>/<yyyy>/<mm>/<dd>/
Here we see a terraform snippet creating a bucket and creating a parameter to retrieve bucket attributes for client specific use cases:
provider "aws" {
region = "us-east-1"
}
# Create an S3 bucket
resource "aws_s3_bucket" "raw_data_bucket" {
bucket_prefix = "company-datalake-"
tags = {
Environment = "DataLake"
Purpose = "Raw_Data_Store"
}
force_destroy = true
}
# Store the bucket name in SSM Parameter Store
resource "aws_ssm_parameter" "bucket_name_param" {
name = "/datalake/resources/s3/bucket/${service_name}/raw_data_bucket_name"
type = "String"
value = aws_s3_bucket.raw_data_bucket.id
tags = {
Environment = "DataLake"
}
}Now a client adding data to the bucket and tagging it clearly for downstream data cataloging.
# Initialize AWS clients
ssm = boto3.client('ssm', region_name=region
s3 = boto3.client('s3', region_name=region)
# Get the bucket name from SSM Parameter Store
response = ssm.get_parameter(Name=ssm_param_name)
bucket_name = response['Parameter']['Value']
# Create key based on source and date
today = datetime.utcnow()
s3_key = f'raw/{data_source}/{today.year}/{today.month:02}/{today.day:02}/data.json'
# Example content (normally this could be your data dump or JSON export)
data = '{"id": 1, "name": "Test Product", "price": 19.99}'
# Upload to S3
s3.put_object(
Bucket=bucket_name,
Key=s3_key,
Body=data,
ContentType='application/json',
Tagging='source=mysql-db&environment=datalake&domain=product'
)- Data Format Choices
Raw zone often stores files as-is (CSV, JSON, XML, etc.)
Consider compressing files (GZIP, Parquet) to reduce costs and speed up transfers. - Lifecycle Policies
Automate transitions to lower-cost storage classes and define expiration rules to clean up stale data.
resource "aws_s3_bucket_lifecycle_configuration" "transition_to_ia" {
bucket = aws_s3_bucket.raw_data_bucket.id
rule {
id = "transition-standard-to-ia"
status = "Enabled"
filter {
prefix = "" # Applies to all objects
}
transition {
days = 30
storage_class = "STANDARD_IA"
}
# Optionally, you can also add expiration
expiration {
days = 365
}
}
}Secure the Raw Data Zone
Security is critical, especially when ingesting sensitive or regulated data.
Best practices include:
- Encryption: Enable SSE-S3 or SSE-KMS for bucket encryption. Use customer-managed keys for stricter controls.
- Bucket Policies: Restrict access by source IP, VPC, or AWS account.
- Access Logging: Consider using AWS CloudTrail for audit logging
- Private Access: Disable public access to all buckets by default.
Looking Ahead
With a secure and structured raw zone in S3, your data lake is off to a strong start. But the raw data zone alone isn’t useful until it’s loaded and transformed in a compute engine. In Part 2, we’ll dive into how Snowflake interacts with S3 via internal and external stages, and how AWS IAM roles empowers secure data ingestion. If you found this explanation helpful give it a clap 👏 and stay tuned for Part 2.
