Understanding Node.js Streams

Node.js streams are a powerful abstraction for handling large or continuous data flows in a memory-efficient and performant way. Whether you’re building file uploads, video delivery, ETL pipelines, or real-time analytics, mastering streams is essential. This post covers concepts, APIs, common patterns, pitfalls, and production-ready best practices — with code examples you can copy, paste, and use.

What streams are

Streams provide data in sequential chunks rather than all at once. That lets you start processing immediately without holding the entire payload in memory.

The four stream types

• Readable — a data source (e.g., fs.createReadStream, req)
• Writable — a destination (e.g., fs.createWriteStream, res)
• Duplex — both readable and writable (e.g., sockets)
• Transform — a duplex that modifies data (e.g., gzip, encryption)

HTTP requests and responses as streams

In Node.js, req is a readable stream and res is a writable stream. That design enables servers to handle large payloads and many concurrent connections efficiently. Each HTTP request gets its own isolated stream objects, which ensures data integrity and predictable behavior under load.

Backpressure — the most important concept

Backpressure is the mechanism that prevents a fast producer from overwhelming a slow consumer. .pipe() handles backpressure automatically. If you need manual control, use this pattern:

readable.on('data', chunk => {
  const ok = writable.write(chunk);
  if (!ok) readable.pause();         // downstream buffer is full — pause upstream
});
writable.on('drain', () => readable.resume()); // downstream ready — resume

Tip: highWaterMark controls buffer sizes (default ~16KB). Adjust it only after measuring.

Readable modes — Paused vs Flowing

• Paused (default): data flows only when you explicitly request it.
• Flowing: data is pushed to consumers automatically (e.g., when you add a 'data' handler or call .pipe()).

Most of the time, prefer .pipe() or for await to avoid low-level mode handling.

pipeline() — safer than .pipe()

.pipe() is convenient, but pipeline() is better for chaining streams because it centralizes error handling and cleanup.

Get Sachin Nadge’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Callback style:

const { pipeline } = require('stream');
const fs = require('fs');
const zlib = require('zlib');
pipeline(
  fs.createReadStream('in.txt'),
  zlib.createGzip(),
  fs.createWriteStream('out.txt.gz'),
  (err) => {
    if (err) console.error('Pipeline failed', err);
    else console.log('Pipeline succeeded');
  }
);

Promise/async style (Node 15+):

const { pipeline } = require('stream/promises');
await pipeline(readable, transform, writable);

Why prefer pipeline():

• Catches errors from any stream in the chain
• Cleans up all streams if one fails
• Better memory management and cleaner code

Modern consumption — async iteration

You can consume readable streams with for await for cleaner, synchronous-looking code:

for await (const chunk of readable) {
  // process each chunk
}

This pattern automatically handles completion and propagates errors correctly.

Error handling and cleanup

• Always attach error handlers to streams, especially to streams you don’t control.
• If one stream fails, destroy() the others to avoid leaks.
• Use pipeline() or stream.finished() to reliably detect completion.
• For HTTP servers, handle aborted connections:

req.on('aborted', () => {
  // destroy related streams, free resources
});

Memory & performance considerations

• Default buffer (highWaterMark) is ~16KB. Very small or very large buffers can hurt performance.
• Fast producer + slow consumer → buffer growth → memory bloat. Respect backpressure.
• Many concurrent streams means many buffers — monitor memory usage and prefer streaming processing over buffering entire payloads.

Common pitfalls

• Missing error handlers → unhandled exceptions.
• Ignoring backpressure → out-of-memory errors.
• Not destroying streams on aborts → resource leaks.
• Trying to reuse streams across requests — always create new streams per request.

Real-world use cases

• Video streaming / adaptive streaming — chunked delivery, range requests.
• Large-file ETL — row-by-row CSV/JSON transforms.
• File uploads — stream to disk or cloud storage without buffering the whole file.
• Real-time pipelines — continuous data flows between systems, Pub/Sub processing.

Node.js vs Browser streams

Conceptually similar, but APIs differ. Node.js streams are mature and server-focused; the Web Streams API targets browsers. You can interoperate when needed, but server-side apps primarily use Node streams.

Best practices — checklist

1. Prefer pipeline() over raw .pipe() for chained streams.
2. Always handle error on every stream.
3. Enforce upload size limits and validate input early.
4. Let .pipe() manage backpressure when possible; use manual pause/resume only for special cases.
5. Destroy/unpipe streams on aborts and errors.
6. Tune highWaterMark only after profiling.
7. Log lifecycle events (open, close, finish, error) for easier debugging.

Node.js Internal Streams

1. File System (fs module)

• fs.createReadStream() → reads files chunk by chunk
• fs.createWriteStream() → writes files in chunks
• Internally, these use libuv’s asynchronous file I/O:
• File descriptor opened
• Data is read/written in small chunks (~64KB default)
• 'data', 'end', 'drain' events emitted as chunks flow
• Use case: reading/writing large files, logs, uploads, downloads

2. HTTP module

• Node.js HTTP request and response objects are streams:
• http.IncomingMessage (req) → Readable
• http.ServerResponse (res) → Writable
• net.Socket is a Duplex stream
• HTTP module wraps the socket in request/response objects
• Data flows in chunks — backpressure handled via socket buffers
• Use case: handling file uploads, streaming responses, large JSON payloads

3. TCP / Sockets (net module)

• net.Socket is a Duplex stream
• TCP packet received → pushed into stream buffer
• 'data' events emitted
• Writable side writes back → OS socket buffer → TCP packets
• Use case: any network app — chat servers, proxies, streaming apps

4. Child processes

• child_process.spawn() and exec() streams:
• child.stdout → Readable stream
• child.stderr → Readable stream
• child.stdin → Writable stream
• Internally connected to the OS pipes / file descriptors
• Use case: streaming stdout/stderr of long-running processes

5. zlib / compression

• zlib.createGzip(), zlib.createDeflate() → Transform stream
• Reads input chunks
• Compresses/encrypts data in chunks
• Emits output chunks downstream
• Use case: HTTP compression, large file compression without loading all in memory

6. Crypto module

• crypto.createCipheriv() / crypto.createDecipheriv() → Transform streams
• Streams encrypt/decrypt chunk-by-chunk
• Backpressure ensures memory-efficient operation
• Use case: secure file storage, HTTPS streaming

7. Process stdio

• process.stdin → Readable
• process.stdout → Writable
• process.stderr → Writable
• Internally wraps TTY or OS streams
• Use case: CLI apps, piping input/output, logs

8. MongoDB / Database drivers

• Mongoose/MongoDB cursors → cursor.stream() or .cursor()
• Batches data from the database
• Pushes results as readable stream
• Prevents loading entire collection into memory
• Use case: export large datasets, ETL pipelines

9. Timers + async iterators (pseudo streams)

• While not true streams, async iterators (e.g., Readable.from()) allow streaming arrays or async data sources
• Under the hood, Node.js uses Readable stream machinery for this

10. Other internal uses

• HTTP/2 module → headers and data frames are streamed
• WebSockets (ws module) → Duplex stream wrapping TCP
• TLS/HTTPS → TLSSocket is a Duplex stream
• Child pipes → pipe() internally uses stream logic with backpressure