Clusterfy — A NodeJS library for simplified process management and IPC

Background: I’m working on an application that needs to be started as cluster. So I started to use the NodeJS core library “cluster”. Later I found out that that I need a shared storage among primary and workers and a way to send unique messages between these ones. I searched for libraries that support both — without success. That’s why I built Clusterfy.

What is Clusterfy?

Clusterfy is an easy-to-use NodeJS library that simplifies creating and managing processes via IPC messaging in a NodeJS cluster. It’s based on the NodeJS core library “cluster” and uses third-party packages like uuid for creating unique IPC messages and rxjs to allow easy event handling. You can find the repository here.

Features

• Simple API for creating, watching and controlling workers in a cluster.
• Shared storage managed by primary process. Workers can set and retrieve this storage.
• Simple API for sending commands between primary and workers: primary <-> worker or worker <-> worker
• Easy way to add custom commands
• Simple event handling using RxJS (e.g. using Observable)
• Extended workers with more attributes like name and status
• Graceful shutdown on process signals
• ES Module with Typescript definitions

Applications

You can use Clusterfy in all NodeJS applications with Node >= 16 and servers like ExpressJS or NestJS.

Limitations

Currently Clusterfy can only communicate with processes started by the current NodeJS application using “cluster.fork”. It can’t communicate and control other processes.

Get Julian Pömp’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Communication works only via messaging.

How it works

The primary process is the manager who knows everything about the workers and controls these. Furthermore the primary process manages a shared storage (JS object). If worker want to retrieve or save values to the storage they send requests to the primary process which returns or saves the value.

Commands from primary process to worker process and vice versa are sent via IPC as ClusterfyIPCEvent objects. It’s also possible to send commands from one worker to another. This works thanks to the primary process that redirects the messages from one worker to another. With this construct you can send commands asynchronously between different processes.

Installation

npm install clusterfy --save

Furthermore, you need to install rxjs >= 7.5.0 and uuid >= 7.0.3 if not already installed.

Usage

Clusterfy is a static class that can be called everywhere. At the entrypoint of your application you want to start the cluster. So you need to initiate the primary process and its workers like that:

import {Clusterfy, ClusterfyIPCEvent, ClusterfyStorage} from 'clusterfy';

const onShutdown = async () => {
    console.log(`Simulate cleanup on shutdown for ${Clusterfy.currentLabel}...`);

    let j = 0;
    for (let i = 0; i < 100000; i++) {
        j++;
    }
    console.log(`All cleaned up for ${Clusterfy.currentLabel} OK`);
};

async function main() {
    if (Clusterfy.isCurrentProcessPrimary()) {
        // initStorage shared memory
        const sharedMemory = new ClusterfyStorage({
            test: {
                some: 1,
            },
        });
        Clusterfy.initStorage(sharedMemory);

        // create some workers asynchronously
        Clusterfy.fork('SomeName1');
        Clusterfy.fork();
        // now create worker that is revived automatically after it died
        Clusterfy.fork('EmailServer', {revive: true});
        await Clusterfy.initAsPrimary({
            gracefulOnSignals: ['SIGINT', 'SIGTERM'],
        });
        Clusterfy.registerShutdownMethod('default', onShutdown);

        // now you can use Clusterfy on primary as you like
        Clusterfy.events.subscribe({
            next: (event: ClusterfyIPCEvent) => {
                console.log(`Primary got event ${event.type} from worker ${event.senderID}`);
            }
        });
    } else {
        await Clusterfy.initAsWorker({
            gracefulOnSignals: ['SIGINT', 'SIGTERM'],
        });
        Clusterfy.registerShutdownMethod('default', onShutdown);
        // worker retrieved metadata from primary on initialization
        console.log(`Worker ${Clusterfy.currentWorker.name} is ready.`);

        // now you can use Clusterfy on worker as you like

        Clusterfy.events.subscribe({
            next: (event: ClusterfyIPCEvent) => {
                console.log(`Worker ${Clusterfy.currentWorker.name} got event ${event.type} from worker ${event.senderID}`);
            }
        });
    }
}

main();

API

There are more functions that you could use. If you want to create your own IPC command, you just have to create a class and extend it with class “ClusterfyCommand”. For example you could create a command “send_mail” and send it to the process that controls an E-Mail queue and sends the mail.

You can find more information the API of Clusterfy here:
https://github.com/julianpoemp/clusterfy#getter

Future

The project could be further improved. For example it could support IPC via network protocols like TCP and UDP and so support communication with processes outside of the current application.