We built a chat app on S3. Three things surprised us.
Customers keep asking us the same question: can I build a chat app on S3? So this week we built one on CDNTR, our CDN and cloud platform, and wrote down what actually happened. The full walkthrough with code is on our site; this post is about the three things that surprised us on the way.
The setup was deliberately small: a Node.js app with Express and Socket.io running as a managed container app, deployed from its source folder with one cdnctl deploy; a managed Postgres for messages; and a bucket on our S3-compatible object storage for attachments. The browser talks to the app over a WebSocket that goes through the CDN, and files go straight between the browser and the bucket.
The demo in under a minute and a half:
S3 is the wrong home for the messages themselves
Object storage is excellent at files: photos, voice notes, PDFs, archives. It cannot do the three things a chat needs most. It never pushes anything, so other clients would have to poll. It cannot append, so every message becomes its own tiny object or a rewrite of a bigger one. And it cannot answer “the last 50 messages in this room” or “what is unread”, because listing keys by prefix is all it knows.
So the honest answer is yes, with S3 as one of three layers: a small WebSocket server that fans out messages, a database that stores them as rows, and object storage for everything that is a file. The app server stays tiny because it never streams media; it signs short-lived URLs and moves small JSON messages.
A 403 that looked like a key problem
The first browser upload failed with 403 AccessDenied on a presigned PUT URL that was, as far as we could see, signed correctly. The same code works on AWS.
The cause: our object storage runs on Ceph RGW, and RGW is stricter than AWS about one header. If the browser sends a Content-Type that is not covered by the signature, RGW refuses the request. AWS simply ignores an unsigned Content-Type, so code written against AWS never notices.
Get Hilmi Baycan’s stories in your inbox
Join Medium for free to get updates from this writer.
The fix is two lines: sign the type into the URL, then send exactly that type.
// server (AWS SDK v3) const cmd = new PutObjectCommand({ Bucket, Key, ContentType: type }); const url = await getSignedUrl(s3, cmd, { expiresIn: 300, signableHeaders: new Set(['content-type']), }); // browser: the same type the server signed await fetch(url, { method: 'PUT', headers: { 'Content-Type': type }, body: file });Two details keep it reliable: use the same fallback on both sides for files with no type (for example application/octet-stream), and don't let an upload helper add ; charset=utf-8 behind your back, because then the header no longer matches the signature. The object storage guide covers keys, buckets and the AWS CLI setup.
WebSockets through a CDN are fine, if they talk
We half expected trouble running Socket.io through the CDN edge. There was none: Socket.io reported websocket as its transport. What matters is silence, not length. On our container app domains the edge closes a WebSocket after 300 seconds without traffic (a delivery rule can set anything from 1 to 3,600 seconds), and Socket.io's own heartbeat every 25 seconds keeps a quiet room well inside that.
Connections still drop: phones switch networks and edges reload their configuration. So the client reconnects on its own, sends join again, and merges the room's recent history by message id, so nothing appears twice.
The checklist we ended up with
- A CORS rule on the bucket for your app’s origin, PUT and GET, and the Content-Type header.
- Content-Type signed into the PUT URL, with one fallback on both sides.
- Presigned URLs that expire in minutes, not days.
- Files never pass through the app; S3 keys never reach the browser.
- Downloads routed through the app, so it can check the user may see that room.
- Reconnect, rejoin, and merge history by id.
The complete guide, with the cdnctl setup, the Postgres table, the Socket.io handler, the CORS rule and the archive pattern for old messages, is here: building a chat app with S3 object storage.
- Hilmi Baycan, CDNTR
Originally published at https://cdntrhq.substack.com on November 6, 2026.
