The FTP Execution Engine, Line by Line: Staging, Validation, and the Atomic Swap
## The one program that actually moves the file — and why production is always the last thing it touches
— -
In [the last poston IBMi FTP process], I walked through why I replaced a pile of ad-hoc FTP CL programs with a single framework: one config table, one execution engine, one housekeeping job. I sketched the execution engine with generic pseudocode and moved on, because the post was already long and the point I was making was architectural, not line-by-line.
When people ask “okay, but what does the program actually look like” that it’s worth a second post. This one has no pseudocode in it. Every gist below is the real shape of the program that runs, on a schedule, for every FTP interface in the framework — genericized names, same logic.
— -
## The data this program touches
Before the code makes sense, you need the shapes it’s working with. Three things:
**A config row**, one per interface — host, credentials (encrypted, see the credentials post), remote path, staging file, production file, a record length for auto-creating either file if it’s missing, max allowed variance percent, allow-empty flag, and the record count from the last *successful* run.
**A history row**, opened at the start of every run and closed at the end — interface, run ID, state (running / complete / failed), start and end timestamps, reason code if it failed.
**Two physical files on the IBM i side** for each interface: the staging file (always cleared before use, always the FTP target, never anything else) and the production file (never written to except by the final copy). There’s no separate archive library — the “previous version” of whichever file is about to be overwritten gets archived as a new member on that same file, not copied somewhere else. That turns “go back to yesterday’s data” into picking a different member on a file you already know the name of, instead of hunting through another library.
That’s the whole shape. Nothing about it is exotic. The discipline is entirely in the order operations happen in, which is what the rest of this post is actually about.
<script src=”https://gist.github.com/jaykrshna/1ee6be6ff72938ff0d4597240c8a5010.js"></script>
— -
## Opening the run
The first thing the engine does, before it touches FTP or the filesystem at all, is write a running row. This has to happen before anything else, because it’s the thing that makes a crashed job detectable later — a row stuck in running with no matching complete or failed is, by definition, a job that died without telling anyone.
The run ID is a plain date-time stamp plus the interface letter, not a sequence number and not a random value — deliberately readable, and, as it turns out, useful for more than logging. Keep this in mind; it comes back later in a way that’s more interesting than “it’s a unique key.”
— -
## Reading the config
One row, one query, into a data structure shaped like the config table. Nothing in it ever gets logged, displayed, or written anywhere except back into local variables. The decrypted password exists only long enough to log in during the transfer step, and gets cleared explicitly the moment that call returns — same rule as the credentials post, applied at every call site, not just the one I wrote about there.
— -
## Auto-creating the files it needs, if they’re missing
The first run against a brand-new interface shouldn’t require someone to pre-create two physical files by hand before anything works. A `CHKOBJ` command run through the database’s command-execution service doubles as an existence check — a non-zero return code means the object isn’t there. If the config row carries a record length, the engine creates the missing file on the spot with `CRTPF`; if it doesn’t, the run stops here with a reason that says exactly what’s missing, instead of failing three steps later with a much less obvious error. The same check runs against the staging file.
— -
## Archiving before anything risky happens
This is the step that makes every failure after this point recoverable instead of catastrophic. Before the staging file is touched, before FTP even runs, whatever’s currently sitting in the file that’s about to be overwritten gets archived — as a new member on that same file, not a copy somewhere else. Adding a member and copying the current data into it means the archive is a snapshot living right next to the data it’s protecting, addressable by a member name, not a filename in some archive library nobody remembers the naming convention for.
Get Mjayakrushna’s stories in your inbox
Join Medium for free to get updates from this writer.
If this step fails — disk full, authority problem, member limit reached — the program stops here. Staging hasn’t been touched. Production hasn’t been touched. Nothing downstream has any idea a run was even attempted.
— -
## Running the actual transfer
This part is intentionally boring, because the interesting design decision already happened: don’t write your own FTP client logic. The engine binds against an FTP API service program and calls its exported procedures directly — open a session, log in, get the file, quit.
A few things worth calling out:
- **The decrypted password gets cleared immediately after the login call returns**, not at the end of the procedure, and not conditionally on success. Whether the login worked or not, the plaintext’s job is done the instant that call comes back.
- **The download targets a fully-qualified IFS path into the staging member**, not a stream file. On IBM i, a path into `/qsys.lib/…/ .file/…/ .mbr` is a normal filesystem path into a database file member — the FTP API doesn’t need to know it’s talking to a table instead of a flat file.
- **There’s no custom retry, no custom protocol handling, no hand-rolled command sequencing** anywhere in this step. That’s deliberate. FTP client behavior is exactly the kind of thing worth getting from a maintained library instead of reimplementing — the framework’s value is everything wrapped around this call, not the call itself.
<script src=”https://gist.github.com/jaykrshna/f6b4bbc5374f636fa9d184f462166ae4.js"></script>
— -
## Validating before anything in production changes
FTP finishing without an error doesn’t mean the data is right. This is the check that actually protects production: count what landed in staging, and compare it against two things.
First, is it zero when it shouldn’t be? Second, has it dropped more than a configured percentage against the last successful run? Say the warehouse feed brought in 300,000 rows last night and the configured variance threshold is 50%. Tonight it brings in 140,000. That’s a 53% drop — past the threshold — so the run halts here with a reason string that records the exact before/after counts, not just “failed.” Nobody has to guess what tripped it; the row says it.
This check runs *before* production is touched, which means a bad upstream batch on someone else’s system stops at the staging file and never gets the chance to overwrite something that was correct.
One more check, worth adding once the first two exist almost for free: if this interface’s data genuinely doesn’t change every run — a reference table, a slow-moving price list — an exact match against the last successful count is a cheap signal that tonight’s file is identical to last night’s, not new data that happens to be the same size. It’s gated behind a flag per interface, not a default, because for most feeds an unchanged count from one night to the next is exactly what a real problem looks like, not evidence of nothing happening. Where it is turned on, it turns “processed the same file twice” from a silent waste of a production overwrite into a one-line skip logged as a success, not a failure.
— -
## The only step that touches production
Everything up to here has been building toward one command: copy staging over production, update the baseline record count for next time, and close the history row as complete. One copy command, one baseline update, one history close. Everything before this point exists to make sure this exact moment is the only time production data changes — and that by the time it runs, the data has already been counted, compared against last night, checked against duplication, and proven not to be garbage.
Every exit from the program, success or failure, goes through the same closing procedure. There’s no path where a run just stops, and the history table doesn’t know about it. If the job itself gets killed mid-run — someone ends it, the subsystem goes down — that’s the one case that procedure can’t catch. That’s exactly the case a follow-up piece on this framework’s locking and crash-recovery logic covers: because the run ID is a plain date-time stamp, not a random value, a row still stuck in running is trivially the oldest running row for its interface the next time anything checks — which turns out to be enough to detect it without any arbitrary timeout at all.
— -
## What this buys you, concretely
Walk a bad night through this: the upstream system on the other end has a bug and only writes half its usual output. The old CL script would FTP that partial file straight into production, the ERP job would run against it at 4 a.m., and the warehouse would spend the day building pick lists from data that was quietly wrong. Nobody would know until someone downstream noticed something was off — hours later, several systems away from where the actual problem started.
With this version: the file lands in staging, the count comes back roughly half of last night’s, the variance check fires, production is exactly as correct as it was yesterday, and the history table has one failed row with a reason string that tells you precisely what tripped it before anyone had to go looking. The bug on the other system is still a bug. It just isn’t your outage anymore.
None of the individual pieces here are clever. A handful of native commands, a couple of calls into a bound FTP library, a couple of count checks. The only thing that took real thought was the order — auto-create, archive, stage, validate, and only then touch production, with a history row open the entire time that can prove, after the fact, exactly which of those steps a given run got to before it stopped.
The next post in this series covers the other half of the engine: the SQL-only lock table that keeps two runs of the same interface from racing each other, why it’s deliberately never touched with native record-level I/O, and what happens when retries run out and the system operator gets asked Retry / Skip / Cancel.
— -
*Jaya Krushna Mohapatra is a Warehouse Management Systems Architect focused on enterprise integrations, IBM i modernization, and scalable backend systems.*
