AWS Glue Explained: The Glue That Holds Your Data Lake Together
A conceptual guide for DevOps and cloud engineers stepping into data engineering on AWS.
Every company has data scattered everywhere: CSV exports in S3, transactions in RDS, events streaming in from applications. Before anyone can run a single useful query, two questions must be answered:
1. What data do we have, where is it, and what does it look like?
2. How do we clean it and move it to where it is useful?
AWS Glue answers both. I recently started learning the AWS data analytics stack as a DevOps and SRE engineer, and Glue was the service that made the whole picture click. This article explains it conceptually, without drowning in code.
What is AWS Glue?
AWS Glue is a fully managed, serverless data integration service. You never provision servers or Spark clusters: AWS runs them while your work runs, and you pay only for that time.
Glue has two halves:
• The Data Catalog: a central store of metadata that describes your data.
• The ETL engine: a built-in Apache Spark engine that extracts, transforms and loads your data.
The library analogy
The easiest way to picture Glue is a library:
Library
AWS Glue
Books on the shelves
Your data (files in S3, rows in RDS)
The card index
The Data Catalog: says where each book is and what is inside, but is not the book
The librarian who walks the shelves and writes the cards
The Crawler
The workshop that repairs, translates and rebinds books
The ETL job
The staff timetable
Triggers and Workflows
Keep this picture in mind. Everything below maps onto it.
Half 1: The Glue Data Catalog
The Data Catalog is a persistent technical metadata store. It stores information about your data, never the data itself. There is one catalog per AWS account per Region, and it is compatible with the Apache Hive metastore.
Its real value is centralisation: Athena, Redshift Spectrum, Amazon EMR, Glue jobs and Lake Formation all read the same catalog. You describe a dataset once, and every tool can use it.
The catalog is built from four components.
1. Databases
A database is a logical folder that groups related tables, such as sales_db or hr_db. Despite the name, it holds no data. It is not a database like RDS.
2. Tables
A table is a definition of one dataset. It records:
• the S3 location (always a prefix, never a single file)
• column names and data types
• the file format, such as CSV, JSON or Parquet
• partition keys, such as year and month
For example, the table orders might say: files under s3://lake/cleaned/orders/, Parquet format, columns order_id, amount, order_date, partitioned by year.
When you drop a Glue table, only the definition disappears. The data in S3 stays.
3. Crawlers
A crawler is a program that connects to a data store, scans the data, works out its schema, and creates or updates tables in the catalog automatically.
A crawler run follows five steps:
1. Connects to the source, such as an S3 prefix or a JDBC database.
2. Reads sample data and detects the format using classifiers.
Get Sunilcns’s stories in your inbox
Join Medium for free to get updates from this writer.
3. Infers column names and types.
4. Detects partitions from Hive-style folder names like year=2026/month=11/.
5. Writes or updates the table in the chosen database.
You can run crawlers on demand, on a schedule, or triggered by S3 events so they only scan new files. A schema change policy decides what happens when columns change: update the table, add new columns only, or just log the change.
Without crawlers, you would write every table definition by hand. With them, the catalog keeps up as your data evolves.
4. Connections
A connection is a saved configuration object that tells Glue how to reach a data store. It holds the endpoint, port, credentials, and network details such as the VPC, subnet and security group.
You need connections for JDBC sources like Amazon RDS, Amazon Redshift or on-premises databases. S3 does not need one. Crawlers and ETL jobs share the same connection, so you configure access once.
Best practice: store the password in AWS Secrets Manager and reference it from the connection, rather than typing it in.
Half 2: The Glue ETL Engine
ETL stands for Extract, Transform, Load. It is the work of turning raw data into useful data.
- Extract means reading data from the source. For example, Glue reads raw CSV order files from S3.
- Transform means cleaning and reshaping that data. For example, it fixes data types, drops bad rows, removes duplicates, and joins orders with customer data.
- Load means writing the result to the target. For example, it writes the cleaned orders as Parquet, partitioned by year, into the
cleaned/zone of the data lake.
ETL jobs
A Glue job is the script that does the ETL work. It runs on Glue’s built-in Apache Spark engine, so it processes large datasets in parallel across many workers.
You can build a job in two ways:
• Glue Studio (visual): drag a source, a transform and a target onto a canvas, and Glue generates the PySpark code.
• Script editor or notebooks: write PySpark or Scala yourself.
Glue also offers lighter job types: Python shell for small scripts, Spark Streaming for continuous data from Kinesis or MSK, and Ray for distributed Python.
Capacity is measured in DPUs (Data Processing Units), roughly 4 vCPUs and 16 GB of memory each. You choose a worker type and count, and you are billed per second.
DynamicFrames
Glue adds its own data structure on top of Spark: the DynamicFrame. Unlike a Spark DataFrame, it tolerates messy or changing schemas, such as a column that is an integer in some rows and a string in others.
Useful DynamicFrame transforms include ApplyMapping (rename columns and change types), ResolveChoice (settle type conflicts), DropFields, Filter, Join and Relationalize (flatten nested JSON). When you need full Spark SQL power, convert it with .toDF() and back again.
A typical job in plain words
1. Read the table orders_csv from the Data Catalog.
2. Map amount from string to double and order_date from string to date.
3. Write the result to s3://lake/cleaned/orders/ as Parquet, partitioned by year and month.
4. Update the catalog so Athena can query the new table straight away.
Job bookmarks
Bookmarks let a job remember which data it has already processed, so each run handles only new files. This is how incremental ETL works in Glue.
A common surprise: a job runs successfully but writes nothing. Usually the bookmark has already covered those files. Reset the bookmark, or disable it for a full reload.
Orchestration: Triggers and Workflows
Jobs and crawlers need something to start them and something to chain them together.
Triggers
A trigger decides when a job or crawler starts. There are three kinds:
• Scheduled: a cron expression, such as every night at 2 AM.
• On demand: started manually from the console, CLI or API.
• Conditional: start one job when another succeeds, fails or finishes.
Workflows
A workflow chains crawlers, jobs and triggers into one pipeline you can run and monitor as a unit. A typical one: crawl the raw data, run the cleaning job, crawl the cleaned data, and the data is ready for Athena.
Outside Glue, you can also start jobs with Amazon EventBridge rules (for example, when a new file lands in S3) or orchestrate complex flows with AWS Step Functions. For a DevOps engineer, this is familiar ground: it is CI/CD thinking applied to data.
Wrapping Up
If you remember one sentence, make it this: S3 stores the data, the Glue Catalog describes it, Glue ETL improves it, and Athena queries it.
Coming from DevOps, I found Glue far less intimidating once I saw it as metadata plus managed Spark, wired together with triggers. In the next article, I will cover AWS Lake Formation, which controls who can see which tables, columns and rows in the lake.
Thanks for reading. If this helped, follow me for more practical AWS and DevOps write-ups.
