<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vincent Perez on Medium]]></title>
        <description><![CDATA[Stories by Vincent Perez on Medium]]></description>
        <link>https://medium.com/@vincentperez91?source=rss-d3e66b5bbb7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8mJgVrqP7-oxUepT</url>
            <title>Stories by Vincent Perez on Medium</title>
            <link>https://medium.com/@vincentperez91?source=rss-d3e66b5bbb7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:42:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vincentperez91/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building the BUS: how we removed the file upload size limit for good]]></title>
            <link>https://medium.com/doctrine/building-the-bus-how-we-removed-the-file-upload-size-limit-for-good-9e0c66c226a9?source=rss-d3e66b5bbb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e0c66c226a9</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Vincent Perez]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 05:58:19 GMT</pubDate>
            <atom:updated>2026-07-24T05:58:19.251Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The story of the Batch Upload Service (BUS), the problem it solved, why we built it the way we did, and why it’s been a quiet success.</em></p><p>By <a href="https://medium.com/u/c6141efdec15">Anatoli De Bradké</a> and <a href="https://medium.com/u/d3e66b5bbb7">Vincent Perez</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtAJ54UhRYZvWBgDhFZYLg.png" /></figure><h3>The problem: “File too large”</h3><p>For a long time, uploading files to our platform meant sending them straight through our Node.js backend. The backend didn’t just relay bytes, it did real work on every file:</p><ul><li>validate the file type and check the extension actually matches the content,</li><li>compute a hash for integrity and deduplication,</li><li>extract attachments out of email files and store them on S3,</li><li>detect duplicates within the batch <strong>and</strong> against everything uploaded before,</li><li>write the right references into the database.</li></ul><p>That works fine for a handful of small files. It falls apart at scale. Routing every byte through the backend meant:</p><ul><li>a hard <strong>file size limit</strong>, because the request had to fit through the backend,</li><li>backends tying up connections and memory, streaming large files instead of serving requests,</li><li>the whole thing getting slower and more fragile exactly when a user uploaded a lot, or uploaded something big.</li><li>processing unvalidated files in the same memory space that handles all core API traffic meant a single malicious upload or a resource-exhaustion attack could blast-radius the entire platform.</li></ul><p>The user-facing symptom was one of the least favorite error message on the platform: <strong>file too large</strong>. The engineering symptom was a single choke point that got worse under load.</p><p>We wanted to remove the size limit entirely and we couldn’t do it without rethinking the pipeline.</p><h3>Why we built a new service</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmtnmFr6KY2fwE2BCDlJvw.png" /><figcaption><strong><em>The “Before” architecture:</em></strong><em> Simple in theory, but concurrent large file uploads quickly choke backend resources and create a single point of failure.</em></figcaption></figure><p>The fix for the size limit is well known and “boring” in the best way: <strong>pre-signed URLs</strong>. Instead of streaming files through the backend, the backend hands the frontend short-lived, pre-signed POSTs, and the frontend uploads <strong>directly to S3</strong>. No backend in the byte path, no request-size ceiling. It’s the industry-standard answer and the most scalable one available.</p><p>But it comes with a catch: if files no longer pass through the backend, <strong>all that per-file processing loses its home.</strong> Validation, hashing, attachment extraction, dedup, database writes: none of it can run “during the upload” anymore, because there is no upload request to hang it on.</p><p>So the real question wasn’t “how do we upload bigger files.” It was:</p><blockquote><strong><em>Once processing moves out of the backend, how do we orchestrate it reliably?</em></strong></blockquote><p>That question is what justified a whole new service instead of a patch. We needed something that could:</p><ul><li>wait for every file in a batch to actually land in S3,</li><li>run per-file processing in parallel,</li><li>run a batch-level step once all files are done (cross-file dedup, consolidation),</li><li>persist results,</li><li>survive partial failures, retries and timeouts,</li><li>and give us visibility when something goes wrong.</li></ul><p>That’s an orchestration problem. So we built an orchestrator: the <strong>Batch Upload Service</strong> (aka BUS).</p><h3>The architecture</h3><p>We chose <strong>AWS Step Functions</strong> as the orchestrator, with the actual work split into small, single-purpose <strong>Python Lambdas</strong>. Step Functions handles the “what runs when, and what happens on failure”; the Lambdas handle the “do the thing.”</p><p>The flow, end to end:</p><p><strong>1. Initiation.</strong> The frontend asks the backend for permission to upload a batch. The backend returns pre-signed POSTs, records the upload intent in the database, and <strong>starts the Step Functions execution</strong> for the batch.</p><p><strong>2. Direct upload.</strong> The frontend uploads files straight to S3 using the pre-signed POSTs. The backend is completely out of the byte path.</p><p><strong>3. Orchestrated processing.</strong> The Step Functions state machine takes over. It:</p><ul><li>waits for each file to be uploaded (more on how below),</li><li>runs a Map state to <strong>process and validate each file in parallel</strong> (up to 40 concurrent),</li><li>runs a single <strong>consolidate</strong> step across the whole batch (this is where cross-file dedup happens),</li><li>runs a second Map state to <strong>persist</strong> each result (up to 20 concurrent).</li></ul><p><strong>4. Status.</strong> The frontend polls the backend for progress and final results. As a safeguard, a <strong>minimal ownership check in Redis</strong> ensures only the user who started an upload can read its status.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_S-z1citAsHB3Ekz1OTCA.png" /><figcaption><em>The AWS Step Functions state machine executing the parallel file processing and batch consolidation.</em></figcaption></figure><h3>The nice bit: waiting for uploads without polling</h3><p>The tricky part of this whole design is step 3’s “wait for the file to be uploaded”. The file goes to S3, but the Step Functions has no idea when it arrives. The naïve solution is to poll S3 in a loop. We didn’t want to pay for that or reason about it.</p><p>Instead, we used Step Functions’ <strong>task tokens</strong> with an event-driven callback:</p><ol><li>For each file, the Map iteration sends an SQS message containing a <strong>task token</strong> and pauses (waitForTaskToken). The iteration is now frozen, costing nothing, waiting for a signal.</li><li>A small Lambda drains that SQS queue and stores each {s3_key → task_token} pair in <strong>DynamoDB</strong>.</li><li>When the file actually lands in S3, S3 emits an event. Another Lambda picks it up, looks up the matching task token in DynamoDB by s3_key, and sends <strong>task success</strong> back to Step Functions.</li><li>That unblocks the frozen Map iteration, and processing for that file begins.</li></ol><p>No polling, no busy-waiting, the workflow sleeps until the file’s own arrival wakes it up. If a file never shows up, a heartbeat/timeout catches it and records an upload_timeout result instead of hanging the whole batch.</p><h3>Another nice bit: dodging payload limits</h3><p>Step Functions passes state between steps as JSON, and there are hard size limits on that payload. When a batch has a lot of files, the inputs and outputs blow right past those limits.</p><p>The fix is the same trick we used for the uploads themselves: <strong>keep the big data out of the pipe.</strong> The Lambdas read and write their payloads directly to a dedicated S3 bucket and only pass <strong>S3 keys</strong> through the state machine. The state machine stays small and fast; the data lives in S3. As a bonus, this is good for confidentiality, since raw, sensitive file data never actually enters the state machine. So it never leaks into AWS execution logs all while preserving a great developer experience.</p><h3>The Lambdas</h3><p>Each Lambda is small, self-contained, and independently deployable (isolated dependencies via uv):</p><ul><li><strong>process_and_validate_file</strong> per file: MIME/extension validation, hashing, metadata extraction, email attachment extraction. Runs under a hard timeout so one pathological file can&#39;t stall an iteration.</li><li><strong>consolidate_files</strong> batch-level: gathers all per-file results and does cross-file work like duplicate detection.</li><li><strong>persist_file</strong> writes results, with <strong>feature-specific</strong> persistence (chatbot, contract analysis, private clause library, flow, extract…). One pipeline, many product features plugging into it.</li><li><strong>store_batch_upload_task_tokens</strong> and <strong>process_batch_upload_s3_events</strong>, the two halves of the event-driven &quot;wait for upload&quot; mechanism above.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JOf1fnKStZqR2mx3VGT-w.png" /><figcaption><strong><em>The “After” simplified design:</em></strong><em> More components to orchestrate, but it permanently solves our scaling issue by letting S3 and AWS Step Functions handle the heavy lifting.</em></figcaption></figure><h3>By the numbers</h3><p>The BUS isn’t a proof of concept sitting in a corner, it’s load-bearing. A 30-day snapshot from Datadog on our production environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBcdkSOT0D55DUpjcaiWvg.png" /></figure><p>The number that best captures <em>why the design works</em> is buried in there: at the Lambda level the error rate is <strong>~0.32%</strong> (about 4,000 transient failures across 1.24M invocations), yet the <strong>end-to-end</strong> workflow success rate is <strong>99.75%</strong>. That gap is Step Functions’ declarative retries doing their job, transient Lambda hiccups get retried and never surface to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAZ1mL61Nw3CKL8MzmdJPQ.png" /><figcaption><strong><em>Datadog Step Functions Metrics Overview:</em></strong><em> Monthly metrics for the </em><em>batch-upload-service highlighting the cyclical daily traffic spikes.</em></figcaption></figure><h3>Why it’s a success</h3><p><strong>It solved the actual problem.</strong> The file size limit is gone. Files go directly to S3, so the ceiling is S3’s, not our backend’s. The error message that started all this is retired.</p><p><strong>The backend was freed up for its actual job.</strong> Large uploads no longer tie up backend connections and memory. The backend hands out pre-signed URLs and starts a workflow instead of streaming gigabytes.</p><p><strong>It scales by design.</strong> Parallel Map states process files concurrently (40 for processing, 20 for persistence), and the event-driven waiting means idle files cost nothing while they wait. Bigger batches just fan out wider.</p><p><strong>It’s resilient without custom plumbing.</strong> Retries, backoff with jitter, timeouts and error handling are declared in the state machine, not hand-rolled in every Lambda. A single bad file degrades to a recorded upload_timeout instead of taking down a batch.</p><p><strong>It’s genuinely debuggable.</strong> Every execution is a visual graph (similar to the state machine diagram above) with the exact input and output of every step. When something fails, you can see <em>which</em> file, <em>which</em> step, and <em>why,</em> which is a large part of why this has been low-drama to operate.</p><p><strong>One pipeline, many features.</strong> Because persistence is feature-pluggable, several product features (chatbot, contract analysis, clause library, flow, extract) ride the same BUS instead of each reinventing upload-and-process.</p><p><strong>Cost efficiency.</strong> While the Step Functions pricing model charges per state transition ($0.025 / 1,000), our total operational bill stays under $40 a month for our current volume. For that price, the amount of custom infrastructure we <em>didn’t</em> have to write, maintain, and debug makes it incredibly cost-effective.</p><h3>The trade-offs we accepted</h3><p>We’re not going to pretend it’s free:</p><ul><li><strong>The reality of tight AWS lock-in.</strong> By choosing Step Functions, we fully committed our architecture to the AWS ecosystem. The entire workflow is declared in Amazon States Language (ASL), and the architecture relies on a tight, proprietary coupling of S3, SQS, DynamoDB, and Lambda. If we ever decided to migrate to another provider, we wouldn’t just be porting Python code; we would have to entirely re-engineer the orchestration layer from scratch. We accepted this platform dependency because the development speed and operational peace of mind were worth it, but it is a strong vendor lock-in.</li><li><strong>Local development.</strong> Running Step Functions locally isn’t as simple as docker run. It needs specific tooling and setup.</li><li><strong>Learning curve.</strong> ASL is its own language, and the team has to ramp up on it, but since we implemented the BUS, it didn’t need much intervention.</li><li><strong>Scaling the upload isn’t scaling everything.</strong> The BUS happily ingests a 500 MB PDF now, but the app still has to <em>display</em> it afterwards, and rendering a file that big is its own, separate problem. We removed the upload ceiling; we didn’t magically make huge files pleasant everywhere downstream.</li></ul><p>We took these on deliberately. The alternative (a hand-built orchestrator with hand-built retries, error handling, and monitoring) is more code we’d own forever, to reimplement things AWS already does well.</p><h3>Wrapping up</h3><p>Building the BUS wasn’t about finding a groundbreaking new way to upload bytes, pre-signed URLs have been the industry standard for years. Instead, it was an exercise in reliable orchestration. By accepting a bit of architectural complexity and leaning into event-driven design, we managed to trade intermittent execution errors for a resilient system with more than 99% success rate.</p><p>Today, the “file too large” error is officially retired, our backend is free to do its actual job, and we have a single, pluggable pipeline powering multiple features across the platform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e0c66c226a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doctrine/building-the-bus-how-we-removed-the-file-upload-size-limit-for-good-9e0c66c226a9">Building the BUS: how we removed the file upload size limit for good</a> was originally published in <a href="https://medium.com/doctrine">Inside Doctrine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>