Meenu and the Table That Remembered
A tale from the Land of Amazonia — Apache Iceberg snapshots, time travel and restore
The screenshots in this tale are real output from the lab manual. Each caption names the step it belongs to, so you can find it again when you run the lab yourself.
TL;DR — Apache Iceberg tables never overwrite. Every write leaves the old version intact, which means a bad write is undoable with SQL and no backup. Here’s a duplicate-load incident, diagnosed and restored, told as a bedtime story.
The hole in the page
By bedtime, Meenu had erased the same quadratic equation so many times that she could see the next page through it.
She held her notebook up to the light. “I think my first answer was right.”
Aparna peered over her glasses. “What was your first answer?”
“If I knew that, I wouldn’t be looking through the page.”
“Go to bed. You can try again in the morning.”
Somewhere between worrying about the algebra test and dreaming that erasers should come with a reverse gear, Meenu fell asleep. Her bedroom ceiling dissolved into stars, and she slipped quietly into her dream land of Amazonia.
Garuda brings excellent terrible news
The Amazonians were already gathered at the Stone Table when a great shadow swept the clearing. Garuda descended through the clouds, wings burning gold, dropped a parchment scroll, circled once above Meenu’s head, and vanished into the night.
Amazon SNS hurried forward, his megaphone head crackling. “WONDERFUL NEWS FROM PLANET EARTH! A SHOP’S REVENUE HAS QUADRUPLED!”
The amphitheatre erupted in cheers.
Meenu reached for the bottom of the scroll. “There’s another line.”
“WITHOUT ANY ADDITIONAL SALES.”
The cheering stopped. A small robot quietly swallowed its party whistle.
The scroll told the rest: a nightly orders job had repeated its work, the dashboard was counting the same purchases again and again, and the finance director was already composing a very enthusiastic email. The engineers had paused the job. Now they needed the correct table back, and proof of the method on a small practice table first.
“Nothing was lost,” said Amazon S3, waddling out of the crowd, his round bucket body rattling with carefully stored treasures. “Everything that shop ever wrote is still inside me. Including the orders they want back.”
He folded his little metal arms. “Somebody only has to ask for the right ones.”
The keeper of earlier pages
A tall bronze robot raised a silver stylus. “Amazon Athena. Ask me a question in SQL and I’ll examine the data.”
Beside her, a smaller robot hugged an enormous card index. “AWS Glue. My Data Catalog tracks every table and where its metadata lives.”
Between them waited a traveller in a frost-edged cloak, holding a book with a golden bookmark. “Apache Iceberg. I’m not a service you open anywhere in the AWS console; I’m an open table format, a way of organising a table that tools like Athena understand.”
He set the book on the desk. Its cover read ORDERS.
“Each time anyone changes the table, this book grows one new page: the table as it stands after that change. Old pages are never torn out. The bookmark marks the page everyone reads; ask the table a question and the answer comes from that page, no other.”
Meenu looked from the bookmark to the earlier pages. “So if someone spoils the table… the spoiled page holds the bookmark. But the good pages are still in the book.”
“Now you know the whole secret. Engineers call each page a snapshot. Our job is to find the right one.”
S3 patted his side. “And every page’s files live with me.”
A very small disaster
Everyone, Meenu included, marched to the Great Workshop, where Athena rebuilt the mistake in miniature: a practice table, five genuine orders. (Every statement they ran is in the lab manual; follow along there.) Five slips floated above the desk. Iceberg turned to a new page and set his bookmark on it.
INSERT INTO toyshop.orders VALUES
(‘ORD-001’,’Priya’,120.00,DATE ‘2026–10–01’),
(‘ORD-002’,’Marcus’,75.50,DATE ‘2026–10–01’),
(‘ORD-003’,’Chen’,210.25,DATE ‘2026–10–02’),
(‘ORD-004’,’Fatima’,42.00,DATE ‘2026–10–02’),
(‘ORD-005’,’Diego’,99.99,DATE ‘2026–10–03’);
“Total revenue, 547.74. All sensible,” Meenu said after she ran a simple SELCT query on the table.
Athena slid the keyboard across. “Now play the villain. The shop’s job ran a statement like this one. It finished the job, took too long to say so, and was sent to do it all over again.”
Meenu read it twice. “It doesn’t bring in any new slips. It reads the table and pours the table back into itself.”
“Which is why no bell ever rang,” Athena said. “The bells only ring when a job fails. This one didn’t fail. It worked, twice, and both times it did exactly what it was told. The data is the only thing that’s wrong. Run it.”
Meenu ran it once, the way the shop’s job had. Five slips became ten. Once more: twenty, because the second repeat copied the duplicates too. Iceberg added two pages, and his bookmark came to rest on the worst of them.
She asked the table to count itself: twenty rows, five distinct orders, revenue 2190.96. Meenu counted on her fingers. “Four copies of every order, and four times the money. Twenty on its own looks reasonable; only the gap between twenty and five gives it away.”
She turned to Iceberg. “Show me when it went wrong.”
He opened the book’s table of contents: every page, when it was written, how many orders it held. Three lines told the story. Five. Ten. Twenty.
Meenu noted the five-order page. Its identifier was an enormous number.
“It’s a name, not a ranking,” Athena said. “Choose your page by its time and its evidence.”
Meenu wrote GOOD PAGE beside it.
Looking through the window
“I want to see what’s actually in that page.”
Get raji krishnamoorthy’s stories in your inbox
Join Medium for free to get updates from this writer.
“You can read an old page without moving the bookmark,” Iceberg said, and held the five-order page up to the light.
Meenu looked through it. Five orders, revenue 547.74, while twenty slips still floated over the desk.
“Nothing changed.”
“Correct,” Athena said. “You read a past version. That’s time travel. An ordinary SELECT still gets the current page.”
Five means five again
Back at the keyboard, Meenu reached for the bookmark. “Can’t we just put this back on the good page?”
“That is the cleanest repair there is,” Iceberg said. “Move the bookmark, write nothing. But you cannot ask me for it from here.”
Athena typed the instruction anyway, so Meenu could see the refusal for herself. “It only runs from Spark — Glue or EMR. In this editor it doesn’t even parse.”
“So what do we do instead?”
“We copy. Clear the current page, then write the good page out again as a fresh page at the end of the book. Same five orders, same ending, two more pages of history.”
Athena wrote. The book grew, the new page held exactly five orders, and the bookmark settled on it.
“That’s the whole rescue?”
“The good page never stopped existing,” Iceberg said. “You made it current again.”
Meenu counted the ordinary way: five rows, five orders, 547.74. She glanced at the book. The troublesome pages were still inside.
“We kept the mistakes?”
“They’re no longer the page anyone reads, but they stay in the book, so you can investigate what went wrong and when. Later, maintenance expires old pages and removes files nobody needs. The past is something you look after.”
S3 pulled his lid down. “And nobody came digging through me for a backup. There wasn’t one. There didn’t need to be.”
Home again
Meenu wrote the rescue on a strip of parchment, three lines in order, and carried it to the Stone Table. Garuda swept down, caught it, and rose towards the stars. A little later a reply drifted down beside her hand:
Ran it on the real table. Orders recovered. Revenue is normal.
Amazon SNS read the last line twice. “SHALL I ANNOUNCE IT?”
“Very quietly,” Meenu said.
The orange sand began to swirl, and the Great Workshop softened at its edges. The last thing Meenu saw was Iceberg closing the book, the golden bookmark resting on the right page at last.
The answer before the hole
Meenu woke with her cheek against her pillow and her maths notebook beside the schoolbag. The hole was still there.
Aparna appeared in the doorway. “Well? Did sleeping on it help?”
“I know how to put an orders table back the way it was before somebody made a mess of it.”
Aparna lowered her glasses. “I was asking about question seven.”
Meenu opened the notebook at a fresh page. In the margin she wrote: Read the history. Check the good snapshot. Copy it forward.
“Can any of this recover the answer you erased?”
Meenu held the damaged page up to the morning light. “Only if this notebook was using Iceberg.”
A single grain of orange sand dropped from her braid. Aparna stared at it.
Meenu turned to question seven.
The story’s words, and the real ones
Every object in Meenu’s dream has a name you will meet in the lab manual:
Lesson learnt
Writes never overwrite. Every INSERT, DELETE or UPDATE on an Iceberg table commits a new snapshot, a complete version of the table, and lays its files beside the old ones. The pointer in the catalog moves; nothing is erased. This is the premise everything else depends on, and it is why the rescue needs no backup.
A successful write can still be a wrong one. The doubling statement committed cleanly, twice. No error, no alarm, a plausible row count. Pipeline health and data correctness are separate things, and only one of them is monitored by default.
The diagnostic is the gap between two counts. COUNT(*) against COUNT(DISTINCT order_id): 20 against 5 means four copies of everything. Twenty rows alone looks fine. That query shape catches duplication anywhere, which is why the manual tells you to keep it.
History is queryable, and recovery runs through it. “db”.”table$snapshots” is a table you read with SQL. Find the last good snapshot, verify it read-only with FOR VERSION AS OF, then restore by emptying the table and re-inserting from its own past. Verify before you restore; one SELECT costs nothing.
The habit to take away: when an Iceberg table looks wrong, read its snapshot history before you change anything.
Join Meenu at the keyboard
Every command in this tale is real, and the attached lab manual runs the whole adventure on your own AWS account in about 30 minutes: Hands-On with Apache Iceberg: Snapshots, Time Travel, and the Undo Button. You’ll need a sandbox account with permissions for Athena, the Glue Data Catalog, S3, and CloudShell, plus an Athena workgroup on engine version 3. The lab is plain SQL in the Athena query editor, no Spark and no clusters.
Go forward deploy
