Stories by Kirill Konshin on Medium

The ultimate Electron app with Next.js and React Server Components

Kirill Konshin — Mon, 21 Oct 2024 07:04:09 GMT

With the emergence of React Server Components and Server Actions writing Web apps became easier than ever. The simplicity when developer has all server APIs right inside the Web app, natively, with types and full support from Next.js framework for example (and other RSC frameworks too, of course) is astonishing.

At the same time, Electron is a de-facto standard for modern desktop apps written using web technologies, especially when application must have filesystem and other system API access, while being written in JS (Tauri receives an honorable mention here if you know Rust or if you only need a simple WebView2 shell).

I asked myself, why not to combine best of both worlds, and run usual Next.js application right inside the Electron and enjoy all benefits that comes with React Server Components?

Demo

I have explored all options available and haven’t found a suitable one, so I wrote a small lib next-electron-rsc that can bridge the gap between Next.js and Electron without running a server or opening any ports.

All you need to use the lib is to add following to your main.js in Electron:

import { app, protocol } from 'electron';
import { createHandler } from 'next-electron-rsc';

const appPath = app.getAppPath();
const isDev = process.env.NODE_ENV === 'development';

const { createInterceptor } = createHandler({
    standaloneDir: path.join(appPath, '.next', 'standalone'),
    localhostUrl: '', // must match Next.js dev server
    protocol,
});

if (!isDev) createInterceptor();

And configure your Next.js build in next.config.js:

module.exports = {
  output: 'standalone',
  experimental: {
    outputFileTracingIncludes: {
      '*': [
          'public/**/*',
          '.next/static/**/*',
      ],
    },
  },
};

Here’s the repository: https://github.com/kirill-konshin/next-electron-rsc and the demo with all files.

And here’s my journey to create this lib.

Motivation to use React Server Components in Electron

Electron’s native way of providing access to system APIs is via IPC or, god forbid, Electron Remote (which was considered even harmful). Both were always a bit cumbersome. Don’t get me wrong, you can get the job done: this and this typed IPC interfaces were the best I found. But with IPC in large apps you’ll end up designing some handshake protocol for simple request-response interaction, handling of errors and loading states and so on, so in real enterprise grade applications it will quickly become too heavy. Not even close to elegance of RSC.

Important benefit of React Server Components in traditional client-server web development has same nature: absence of dedicated RESTful API or GraphQL API (if the only API consumer is the website itself). So developer does not need to design these APIs, maintain them, and app can just and just talk to backend as if it’s just another async function.

With RSC application all logic can be colocated in the Web app, so Electron itself becomes a very thin layer, that just opens a window.

Here’s an example, we use Electron’s safe storage and read from/write to file system right in the React Component:

import { safeStorage } from 'electron';
import Preview from './text';
import fs from 'fs/promises';

async function Page({page}) {
  const secretText = await safeStorage.decryptString(await fs.readFile('path-to-file'));
  
  async function save(newText) {
    fs.writeFile('path-to-file', await safeStorage.encryptString(newText));
  }
  
  return ;
}

Such colocation allows much more rapid development and much less maintenance of the protocol between Web and Electron apps. And of course you can use Electron APIs directly from server components, as it’s the same Node.js process, thus removing the necessity to use IPC or Remote, or any sort of client-server API protocol like REST or GQL.

Basically, this magically removes the boundary between Electron’s Renderer and Main processes, while still keeping everything secure. Besides you can shift execution of heavy tasks from browser to Node.js, which is more flexible how you distribute the load. The only problem is… you need to run an RSC server in Electron. Or do you?

Requirements

I had a few and very strict requirements that I wanted to achieve:

No open ports! Safety first.
Complete support Next.js: React Server Components, API Routes (App router) and Server Side Rendering, Static Site Rendering and Route Handlers (Pages router), you name it, with strict adherence established patterns
Minimal, easy to use, based on standards, basically an enterprise-grade, production ready stack for commercial use, mature and well known set of technologies
Performance

After some research I found an obvious choice called Nextron. Unfortunately, seems like it does not utilize the full power of Next.js, and does not support SSR (ticket remained open in Oct 2024). On the other hand there are articles like this or this, both very close, except for usage of server with an open port. Unfortunately I only found it after I came up with the approach I’m about to present, but the article validated it. Luckily I found it before writing this post, so I can give kudos to the author here.

So I started exploring on my own. Turned out, the approach is pretty simple. And all the tools are already available, I only needed to wire them together in some unorthodox way.

Next.js

First step would be to build Next.js app as a standalone. This will create an optimized build which contains all modules and files that can possibly be required in runtime, and removes everything that’s unnecessary.

module.exports = {
  output: 'standalone',
  experimental: {
    outputFileTracingIncludes: {
      '*': [
          'public/**/*',
          '.next/static/**/*',
      ],
    },
  },
};

Aaand, this is it for Next.js.

outputFileTracingIncludes is needed so that optional public and .next/static folders will be copied to standalone build. Next.js assumes you should publish this to CDN, but in this case everything is local.

Next step is a little trickier.

Electron

Now I need to let Electron know that I have Next.js.

One possible solution is Electron’s Custom Protocol or Schema. Or a Protocol Intercept. I chose the latter as I’m perfectly fine to pretend to load web from http://localhost (emphasis on pretend as there should be no real server with an open port).

Besides, this also ensures relaxed policy of one “popular video service”, that forbids embedding on pages served via custom protocols 😅.

Please note that I purposely excluded a lot of unnecessary code to focus on what matters to show the concept.

To implement the intercept I added following:

const localhostUrl = '';

function createInterceptor() {
    protocol.interceptStreamProtocol('http', async (request, callback) => {
        if (!request.url.startsWith(localhostUrl)) return;
        try {
            const response = await handleRequest(request);
            callback(response);
        } catch (e) {
            callback(e);
        }
    });
}

This interceptor serves static files and forwards requests to Next.js.

Honorable mention here goes to awesome Electron Serve, which implements a custom schema for serving static files.

Bridging Electron and Next.js

Next step would be to create a file to provide some convenience to use the non-existing port-less “server”:

import type { ProtocolRequest, ProtocolResponse } from 'electron';
import { IncomingMessage } from 'node:http';
import { Socket } from 'node:net';

function createRequest({ socket, origReq }: { socket: Socket; origReq: ProtocolRequest }): IncomingMessage {
    const req = new IncomingMessage(socket);
    req.url = origReq.url;
    req.method = origReq.method;
    req.headers = origReq.headers;
    origReq.uploadData?.forEach((item) => {
        req.push(item.bytes);
    });
    req.push(null);
    return req;
}

createRequest uses a Socket to create an instance of Node.js IncomingMessage, then it transfers the information from Electron’s ProtocolRequest into the IncomingMessage, including the body of POST|PUT requests.

import { ServerResponse, IncomingMessage } from 'node:http';
import { PassThrough } from 'node:stream';
import type { Protocol, ProtocolRequest, ProtocolResponse } from 'electron';

class ReadableServerResponse extends ServerResponse {
    private passThrough = new PassThrough();
    private promiseResolvers = Promise.withResolvers();
    constructor(req: IncomingMessage) {
        super(req);
        this.write = this.passThrough.write.bind(this.passThrough);
        this.end = this.passThrough.end.bind(this.passThrough);
        this.passThrough.on('drain', () => this.emit('drain'));
    }
    writeHead(statusCode: number, ...args: any): this {
        super.writeHead(statusCode, ...args);
        this.promiseResolvers.resolve({
            statusCode: this.statusCode,
            mimeType: this.getHeader('Content-Type') as any,
            headers: this.getHeaders() as any,
            data: this.passThrough as any,
        });
        return this;
    }
    async createProtocolResponse() {
        return this.promiseResolvers.promise;
    }
}

ReadableServerResponse is basically just a regular Node.js ServerResponse from which I can read the body once Next.js finishes the processing. createProtocolResponse converts the ReadableServerResponse into Electron’s ProtocolResponse.

createProtocolResponse method returns a Promise which waits for the body and resolves into converted ReadableServerResponse as ProtocolResponse.

Next step is finally the “server” itself.

No server, no ports

import type { ProtocolRequest, ProtocolResponse } from 'electron';

export function createHandler({
    standaloneDir,
    localhostUrl = '',
    protocol,
    debug = false,
}) {
    const next = require(resolve.sync('next', { basedir: standaloneDir }));

    const app = next({
        dev: false,
        dir: standaloneDir,
    }) as NextNodeServer;

    const handler = app.getRequestHandler();

    const socket = new Socket();

    async function handleRequest(origReq: ProtocolRequest): Promise {
        try {
            const req = createRequest({ socket, origReq });
            const res = new ReadableServerResponse(req);
            const url = parse(req.url, true);

            handler(req, res, url);

            return await res.createProtocolResponse();
        } catch (e) {
            return e;
        }
    }

    function createInterceptor() { /* ... */ }

    return { createInterceptor };
}

I use the NextServer from Next.js app’s standalone build to create a handler, a regular Express-like route handler which takes Request and Response as arguments.

Key function here is handleRequest.

It provides a dummy Socket to createRequest to create a dummy IncomingMessage, creates a dummy ReadableServerResponse. I feed both request and response to Next.js’s handler, so Next.js can work its magic, not knowing that there’s no actual server, just dummy mocks. Once handler finishes its job the ProtocolResponse is ready for Electron to send to browser. And this is it.

Note that I don’t actually start the Next.js or any other server anywhere, so Requirement #1 is achieved, no ports are open. You can take a look at Next.js documentation to learn more about regular way of setting up a handler with the server. And since I use regular Next.js way, Requirement #2 is achieved.

And since this whole approach works fine on highly loaded servers, and with Electron there’s just one user at any time, the performance Requirement #4 is achieved as well.

Bundling and publishing

I suggest to use Electron Builder to bundle the Electron app. Just add some configuration to electron-builder.yml:

includeSubNodeModules: true

files:
  - build
  - from: '.next/standalone/demo/'
    to: '.next/standalone/demo/'

For convenience, you can add following scripts to package.json:

{
  "scripts": {
    "build": "yarn build:next && yarn build:electron",
    "build:next": "next build",
    "build:electron": "electron-builder --config electron-builder.yml",
    "start:next": "next dev",
    "start:electron": "electron ."
  }
}

For separation of concerns I recommend to keep Next.js sources in src of and Electron soures in and src-electron, this ensures Next.js does not try to compile Electron.

Conclusion

Requirement #3 is achieved in full glory since it’s just one file, and it only uses standard APIs.

I was amazed when it actually worked… I was quite skeptical that it would be this simple and yet so elegant.

Now I can enjoy full access to file & operating systems directly from Next.js Server Components or Route Handlers, with all the benefits of Next.js ecosystem, established patterns, and while using Electron to deliver the complete app experience to users, since the app can be bundled and published.

P.S. I have done my due diligence and I have not found any articles that cover usage of Next.js with mocked requests and responses, especially in conjunction with Electron. Shame on me if otherwise, I must have forgotten how to Google 🤓… But even if I missed something, this article should help to explain why this approach is good.

P.P.S. MSW is a bit overkill and is used for different purposes, like other HTTP mocking libraries.

P.P.P.S. Few shady things in the code are using buffers to read response and synchronous reading static files, both can be improved with streaming, but for simplicity it’s good enough.

Web Apps: Micro Frontend framework with support of Webpack Module Federation

Kirill Konshin — Mon, 15 Jun 2020 17:13:43 GMT

Web Apps: Micro Frontend framework with support of Module Federation

I would like to present a multi-framework, multi-bundle orchestration layer, supporting the oldest to the newest technologies of micro frontends. Framework leverages the power of Webpack to work hand in hand to coordinate and route multi-bundle applications without the overhead.

These days micro frontends are hot and trendy. Here’s how Martin Fowler defines them:

In short, micro frontends are all about slicing up big and scary things into smaller, more manageable pieces, and then being explicit about the dependencies between them. Our technology choices, our codebases, our teams, and our release processes should all be able to operate and evolve independently of each other, without excessive coordination.

This approach becomes crucial when an application outgrows all reasonable limits — the build process is slow, every minor change results in minutes of compilation in watch mode, and the codebase contains thousands of modules and NPM dependencies. At this point, the support of this monster becomes too expensive.

The problem becomes even worse if the team decides to try a new, improved technology stack. You can attempt to gradually replace the old system with the new one. At RingCentral we’ve been through that: we attempted to refactor one monolith and replace it with another monolith. Both were rapidly growing and receiving addition of new features, so when we finished, the final monolith was enormous. At some point it became obvious that it’s time to divide it before it’s too late, so we introduced embeddable sub-apps, but each was developed by a different team and had its own interface to connect to the host, its own bus, and some of them were IFrames. This approach helped in the short term, but was suboptimal in terms of further scaling.

How to plug an app written using one framework into another, incompatible app, that uses different framework? You need an abstract orchestrator!

Before we start, let’s first figure out the basic terms:

Apps can’t talk directly to each other, but host wire them together

A Host is an app that contains a bunch of smaller apps
A Child App, or just an App, is an IFrame App, a Global App, or a Web Component App which we embed in the Host or another Child App (which can act like a Host, too)
An Orchestrator: Web Apps Framework is an agent who helps them talk together

The Web Apps Framework

The Web Apps framework provides exactly this:

A framework agnostic way to inject any app into any app, with a unified communication interface along with other perks.

Links to repository: https://github.com/ringcentral/web-apps and demo: https://ringcentral-web-apps.vercel.app.

Consider this screenshot, taken from the demo:

Arrows are pointing at different apps, that are seamlessly presented together but can be deployed and developed separately. Two of them are IFrames, which is a useful approach in case you need to deal with legacy code, that could be written entirely without JavaScript, or an app that needs maximum isolation of CSS and code.

Of course, there are other frameworks that do similar things, but Web Apps is using a very elegant and minimalistic approach to loading and wrapping different kinds of apps, which provides a unified interface no matter which underlying technologies are being used: Angular, Vue, React, jQuery, no JS at all, you name it.

And Web Apps provides first class support for React, just point to the app’s URL and voila, it will be included in the DOM and all events and handlers will be wired automagically:

import {useApplication, eventType, useListenerEffect, dispatchEvent} from '@ringcentral/web-apps-host-react';

const Page = () => {
    const {Component, node} = useApplication({
        id: 'xxx',
        type: 'script',
        url: [ 
           'http://example.com/styles.css', // yep, with styles
           'http://example.com/bundle.js',
           'http://example.com/entry.js'
        ]
    });

    useListenerEffect(node, eventType.message, (message) => {
        alert(`message from app: ${message}`);
    });

    return <>
        <Component someProp="someValue" />
        
    ;
};

The concept of this framework is to:

Load an App with certain type & URL (one or many)
Render the App on the page in any place
Wire events between Host and the App, including navigation events

Communication can be event-based or through props (or HTML attributes). Events with will be transmitted to any type of app, including IFrame, through the same interface, so the Host can swap them at any time and should not care what type of app it’s showing. The Host can even swap URL and type of an App dynamically, let’s say, based on location, and show different apps in the main content area (just like in the demo).

For other frameworks, Web Component helper can be used:

// somewhere in your app
import '@ringcentral/web-apps-host-web-component';

// then just add this

How to define an app?

There are no special techniques to prepare apps for embedding; just a tiny wrapper, depending on the type of app:

Global Apps — a script that injects something into a given DOM node
Web Component Apps—loaded as script and wrapped in Custom Element, provides a good isolation, thanks to Shadow DOM and Shadow CSS
IFrame Apps — for extra security and isolation, depending on the degree of trust between host and app

Global Apps

Let’s say you have a React app that you’d like to embed. With Module Federation, the entry point will look like just a simple default export of a callback that will inject the app in a given node:

import React from "react";
import {render, unmountComponentAtNode} from "react-dom";
import {registerAppCallback} from "@ringcentral/web-apps-react";
import App from "./App";

export default (node) => {
    ReactDOM.render(, node);
    return () => ReactDOM.unmountComponentAtNode(node);
});

We’ve worked with Module Federation (kudos to Zack Jackson) to make sure Web Apps have seamless integration.

With a bit of additional code using MutationObserver, all attributes can be synchronized from host to app.

If Module Federation is not available on the host inject your app into a given DOM Node, with a JSONP-style callback, as follows:

import React from "react";
import {render, unmountComponentAtNode} from "react-dom";
import {registerAppCallback} from "@ringcentral/web-apps-react";
import App from "./App";

registerAppCallback('%appId%', (node) => {
    ReactDOM.render(, node);
    return () => ReactDOM.unmountComponentAtNode(node);
});

Web Component Apps

Another way is to wrap the app in a Web Component for better isolation of CSS styles using Shadow DOM and Shadow CSS, as follows:

import React from "react";
import {render, unmountComponentAtNode} from "react-dom";
import {App} from './app';

const template = document.createElement('template');

template.innerHTML = `
    
    

`;

customElements.define('web-app-%appId%', class extends HTMLElement {
    constructor() {
        super();
        this.attachShadow({mode: 'open'});
        this.shadowRoot.appendChild(
            document.importNode(template.content, true)
        );
        this.mount = this.shadowRoot.querySelector('.container');
    }

    connectedCallback() { render(
                    authtoken={this.getAttribute('authtoken')}
            node={this}
        />, 
        this.mount
    ) }

    disconnectedCallback() { unmountComponentAtNode(this) }    
});

With observable attributes you can sync them to React app as props.

IFrame Apps

Using IFrames does not require any code at all, but if you want to communicate just add this:

import {
    IFrameSync, dispatchEvent, eventType
} from '@ringcentral/web-apps-sync-iframe';

export const sync = new IFrameSync({
    history: 'html5',
    id: 'iframe',
    origin: window.location,
});

// now you can listen and dispatch:
const node = sync.getEventTarget();

dispatchEvent(node, eventType.message, 'Hello from IFrame');

node.addEventListener(eventType.message, message => alert(message));

What about sharing?

Obviously, if each app begins to have its own build, it also carries all the common dependencies, like react or react-dom. This, at times, can be quite heavy. Luckily, there’s a fancy new technology that aims to solve this problem:

Webpack 5 Module Federation: A game changer in JavaScript architecture

The big win is that the Web Apps framework has out of the box support for Module Federation which allows sharing of modules between host and apps, and apps between other apps. This especially plays well with small footprint of the framework. Module Federation has many features including automatic healing, redundancy and proper versioning. You can just follow the example host and app:

// host's webpack.config.js
module.exports = {
    ..., // all the usual stuff
    plugins: [
        new ModuleFederationPlugin({
            name: 'web-app-host',
            library: {type: 'var', name: 'web-app-host'},
            shared: ['react', 'react-dom'],
        }),
    ]
};

// app's webpack.config.js
module.exports = {
    ..., // all the usual stuff
    plugins: [
        new ModuleFederationPlugin({
            name: 'web_app_federated',
            library: {type: 'var', name: 'web_app_federated'},
            filename: 'remoteEntry.js',
            exposes: {
                './index': './src/index',
            },
            shared: {
                'react-dom': 'react-dom',
                moment: '^2.24.0',
                react: 'react',
            },
        }),
    ]
};

And this is it! Now your host and app can share dependencies in a clean and transparent way.

More about IFrames

Of course, one can just display an IFrame with a legacy app, no big deal. But big problems may occur when sub-apps are contained within IFrames and wish to send and receive messages from the host and show popups.

Messaging can be solved via PostMessage, but Web Apps simplifies and standardizes the interface, making sure there’s no flood of broadcast post messages and that only proper targets will receive them.

Dealing with the popups problem is a bit trickier:

Note the visual separation of app and host

… the gray backdrop behind the popup is contained within the IFrame, while the rest of the website has a white background, meh, ugly!

Can we tell the host that the IFrame’d app is showing a popup? What if the host wants the IFrame’d app to hide it?

This is exactly what the Web Apps Framework is doing: it makes sure the Host is aware when to show the backdrop and what color it should be. It can send an event to the IFrame’d App when it’s time to close the popup, so it looks seamless:

Backgrounds inside and outside of IFrame are the same

Framework also supports synchronization of IFrame height and it’s contents, so that long texts in frames aren’t cut.

Quick Recap

Let me summarize the key features of the Web Apps Framework:

Module Federation support
Location synchronization between app and host
Ability to deep-link “app to app” or “app to host” or “host to app”
Consistent event-based interaction between apps and host
IFrame resizing based on the content of an IFrame
IFrame popup support
Maximum adherence to Web Standards
Written in TypeScript
React and Web Component host helpers
Unlimited nesting of apps within other apps, e.g. each app can become a host for more apps
Seamless navigation support

Check out the repository with examples: https://github.com/ringcentral/web-apps and demo: https://ringcentral-web-apps.vercel.app.

Kudos

Zack Jackson, Chris Van Rensburg and Anton Bulyonov

Web Apps: Micro Frontend framework with support of Webpack Module Federation was originally published in RingCentral Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Speed up NPM/Yarn install in Gitlab

Kirill Konshin — Wed, 22 Apr 2020 08:33:55 GMT

This article is the sum of my findings in scope of Gitlab Issue that’s still unresolved at this moment (April 2020). In short, when the node_modules becomes large Gitlab is experiencing huge performance bottleneck due to great amount of files it needs to archive and upload during caching procedures. I have came up with few ways to improve caching performance.

At RingCentral we use runners and local S3 cache server, we have 120K files after yarn install of one of our monorepos, with overall size about 400–600 megabytes. Monorepo is a git repository which has multiple packages at once, packages can have own dependencies and usually are linked together. This is the worst case scenario since amount of Node modules can become insanely big. But this article can help you to deal with regular repositories too if they have lots of dependencies.

Advice zero would be to use Yarn Workspaces to deal with monorepo instead of Lerna, as latter is 2–3 times slower. Install and caching phases timings roughly are:

Cache download takes about 1 minute
Cache create 4 minutes (zipping of hundreds of thousands files)
Cache upload 1 minutes
Bare Yarn install 3 minutes
Yarn install on top of cache 1 minute

This is our baseline, now let’s analyze how we can speed things up.

Solution 1: Create cache in first job, run others with read-only cache

before_script:
  - yarn install

cache:
  key: $CI_PROJECT_ID
  policy: pull
  untracked: true

install:
  stage: install
  script: echo 'Warming the cache'
  cache:
    key: $CI_PROJECT_ID
    policy: pull-push
    paths:
      - .yarn
      - node_modules
      - 'packages/*/node_modules'

lint:
  stage: test
  script: yarn run lint:all

Key measurements:

Install, cache create & upload: 3+4+1 = 7 min
Download, install, skip upload: 1+1 = 2 min

Best case scenario without cache upload it actually speeds things up a lot. It’s OK to waste time in ONE job to create and upload cache, if you have many other jobs that only download.

Given cache create+upload takes 5min, and Yarn install with cache saves us (3min pure install - 1min cached install) = 2min, it means that at least two jobs must successfully download cache and utilize it in Yarn install to justify time spent on uploading. More jobs use the cache — the better.

Solution 1, but with per-job cache for selected jobs

Down side is that cache can’t be configured “per-job” since you can have only one job that uploads cache. If you need to cache something else besides node_modules you're stuck. And unfortunately, jobs like ESLint spend more time on actual analysis than on installation etc., so caching those artifacts is even more important than node_modules.

Solution for this could be to disable Yarn workspaces for such jobs, only top level packages will be installed, which allowed to cut Yarn install time from 3.5min to ~1min, so no need to cache node_modules at all, just cache ESLint artifacts.

lint:
  stage: test
  before_script:
  - sed -i 's/workspaces/workspaces-disabled/g' package.json
  - yarn install --no-lockfile
  - sed -i 's/workspaces-disabled/workspaces/g' package.json
  script: yarn lint
  cache:  # here we use per-job cache
    key: $CI_COMMIT_REF_NAME-lint
    paths:
      - .eslint

Solution 2: Build a docker image and use it in other jobs

Since each job is running in a container we can build an image for the container as first step. We will be using Kaniko because it is faster than regular Docker-in-Docker (DIND):

stages:
  - docker
  - build
  - test

variables:
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

docker:
  stage: docker
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
  - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"username\":\"${CI_REGISTRY_USER}\",\"password\":\"${CI_REGISTRY_PASSWORD}\"}}}" > /kaniko/.docker/config.json
  - time /kaniko/executor --context="${CI_PROJECT_DIR}" --dockerfile="${CI_PROJECT_DIR}/Dockerfile" --destination="${IMAGE_TAG}" --cache=true

build:
  image: "${IMAGE_TAG}"
  stage: build
  script: yarn build
  artifacts:
    paths:
      - 'packages/*/lib'  # all artifacts that are used by tests
  variables:
    GIT_STRATEGY: none

lint:
  image: "${IMAGE_TAG}"
  before_script: cd /opt/workspace
  script: yarn lint
  variables:
    GIT_STRATEGY: none
  cache:  # here we use per-job cache
    key: $CI_COMMIT_REF_NAME-lint
    paths:
      - .eslint

We’re using Gitlab’s own Container Registry via environment variables. But you can use Docker Hub or anything else too, just define your own ENV variables.

We can skip the git checkout (GIT_STRATEGY: none) since the code is already a part of image.

In order to get maximum optimization we will build a special image manually (as described in the article https://medium.com/@dSebastien/speeding-up-your-ci-cd-build-times-with-a-custom-docker-image-3bfaac4e0479) and push it from local computer to registry once in a while. Dependencies of this base image may become outdated, so on CI we will use base image just as a foundation for real image which will be used in jobs. Pre-warmed cache of the base image will help to speed up build of the second one.

Let’s create a shared.Dockerfile for the base image.

FROM ringcentral/web-tools:alpine

WORKDIR /opt/workspace

ADD yarn.lock package.json lerna.json ./

# Note!
# Add one by one, you can't use glob because of this bug in Docker
ADD packages/a/package.json packages/a/package.json
ADD packages/b/package.json packages/b/package.json
ADD packages/c/package.json packages/c/package.json

RUN yarn install --link-duplicates

You can build and push this image like so:

$ docker login YOUR-GITLAB-OR-OTHER-REGISTRY
$ docker build --tag SOMENAME --file shared.Dockerfile .
$ docker tag SOMENAME SOMENAME:latest
$ docker push SOMENAME:latest

This will require a DOCKERFILE, in which we will do pretty much the same but it would be based on the previous image:

FROM SOMENAME:latest

WORKDIR /opt/workspace

ADD yarn.lock package.json lerna.json ./

# Note!
# Add one by one, you can't use glob because of this bug in Docker
ADD packages/a/package.json packages/a/package.json
ADD packages/b/package.json packages/b/package.json
ADD packages/c/package.json packages/c/package.json

RUN yarn install --link-duplicates  # this cuts size of node_modules

ADD . ./  # now we add all the files

RUN rm -rf $(yarn cache dir)

Make sure to have .dockerignore and ban all node_modules and build artifacts, otherwise your image will become too huge.

In our case timings were:

Install step of docker build on CI: 3 min
Image upload: 1 minute
Image download: 1.5 minutes

Which gives similar time as solution 1: Kaniko build job overall takes 5 min, regular about 6–7 min.

Overall looks like a viable approach for heavy jobs that do require full install. Image download + quick install (1.5 min+1 min) take less time than full Yarn install (3.5 min) or Gitlab cache download, so having image is beneficial.

For quick jobs like ESLint, which does not require full install of monorepo, just top-level, it is an overkill though, since these jobs won’t need all dependencies anyway, but that’s micromanagement…

Solution 3: Mount global Yarn cache to runner & don’t use Gitlab cache for node_modules

Gitlab jobs are running in Docker containers so we can mount a directory used by Yarn to globally cache packages in container to a directory in the runner. This will make any container to always have pre-warmed global Yarn cache. Unfortunately, this saves time on network downloads, but Yarn still copy tons of files into place during install.

To do that we need to add following to config.toml:

# /etc/gitlab-runner/config.toml
[[runners]]
  ...
  [runners.docker]
    volumes = ["/cache:/usr/local/share/.cache:rw"

After that disable caching of node_modules in all jobs.

This produced steady 2 minutes installation time, and now you can cache individual artifacts per-job. You can speed it up by using top-level installs for certain jobs too.

stages:
  - docker
  - build
  - test

image: node:lts

before_script:
  - yarn install

build:
  stage: build
  script: yarn build
  artifacts:
    paths:
      - 'packages/*/lib'  # all artifacts that are used by tests

lint:
  stage: test
  script: yarn lint
  cache:  # here we use per-job cache
    key: $CI_COMMIT_REF_NAME-lint
    paths:
      - .eslint

Solution 4. Yarn 2 with PNP

Yarn 2 with Plug’n’play remediates caching issue dramatically: https://dev.to/arcanis/introducing-yarn-2-4eh1.

You even can use regular cache because it’s still fast enough, Yarn’s cache contains zip files instead of thousands of small sources. And install itself is much faster because Plug’n’play eliminates the time spent on copying from cache to workspace.

This approach requires quite a lot of effort to resolve all the bugs of early software, but as a reward you can use the cache as intended:

stages:
  - docker
  - build
  - test

image: node:lts

before_script:
  - yarn install

# here we can use one cache for all jobs
cache:
  paths:
    - node_modules
    - "packages/*/node_modules"
    - .eslint

build:
  stage: build
  script: yarn build
  artifacts:
    paths:
      - 'packages/*/lib'  # all artifacts that are used by tests

test:
  stage: test
  script: yarn test

Speed is amazing. First time w/o cache it takes same 1–2 minutes, but cache creation and upload take ~20 seconds each. But magic happens on next installs, when cache is warm: install takes 10 seconds (!!!), cache download same ~20 seconds. If we disable cache upload overall freeload on subsequent jobs is just 30 seconds!!!

Summary

As you see, worst performance is regular Gitlab Cache. Even quite fast execution with blocked cache uploads does not justify the performance failure on creation and upload.

Second comes Kaniko. Quite fast overall, faster w/o cached image, but slower on subsequent jobs. Requires juggling with Docker images. Allows to have cache per job.

Simple idea to mount global Yarn cache to runner resulted in surprisingly good and stable results, a bit worse than regular Gitlab cache but it depends how many subsequent jobs you have. If not many then it’s your best bet.

But the absolute winner undoubtedly is Yarn 2 with PNP. It’s much faster on first run, and 30 seconds on subsequent jobs is the best thing ever. And you can make it even faster if you commit it’s local cache ;)

Speed up NPM/Yarn install in Gitlab was originally published in D.i.S on Medium, where people are continuing the conversation by highlighting and responding to this story.

React, JSX, ES module imports (dynamic too) in browser without Webpack

Kirill Konshin — Fri, 17 Jan 2020 19:52:40 GMT

This article is an attempt to put together everything available at the moment and see if it is really possible to implement production ready React-based website without pre-compilation like Webpack/Parcel or at least to have as less pre-compilation as possible.

TL;DR

It is possible to serve React project with minimal compilation of JSX without using bundlers like Webpack or Parcel.

Bundlers pack code into chunks, which has drawbacks: chunks have to be small enough to deliver only what’s needed and big enough to make better use of HTTP1.1 connections. Finding the right balance may be tricky and involves some automatic and manual tuning. In addition it slows down the build and dev mode.

We will use Service Workers to do all the work in runtime and some Babel for production build.

Everything described in the article is highly experimental and I am cutting corners everywhere. By no means I recommend to use any of that in production :)

Current state

Ability to use ECMAScript modules (

Where in src/app/index.jsx we will have:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';

(async () => {
  const {Button} = await import('./Button.jsx');
  const root = document.getElementById('root');
  ReactDOM.render((
    
      
    

  ), root);
})();

And the src/app/Button.jsx:

import React from 'react';
export const Button = ({children}) => ;

Does it work? Of course, no. Even though we’ve successfully imported everything.

Let’s move on to the next challenge.

UMD support

Dynamic way

The issue now is that React is distributed as UMD, it cannot be consumed by imports, even by the shimmed ones (if the ticket is resolved, just skip this step). So we need to somehow patch the distributable to convince browser that it’s a legit ES modules.

The above mentioned article led me to an idea that we can use Service Workers to intercept and pre-process network requests. Let’s create the main endpoint src/index.js, which will bootstrap the SW and App and use it instead of the App directly (src/app/index.jsx):

(async () => {

  try {
    const registration = await navigator.serviceWorker.register('sw.js');
    await navigator.serviceWorker.ready;

    const launch = async () => import("./app/index.jsx");

    // this launches the React app if the SW has been installed before or immediately after registration
    // https://developers.google.com/web/fundamentals/primers/service-workers/lifecycle#clientsclaim
    if (navigator.serviceWorker.controller) {
      await launch();
    } else {

navigator.serviceWorker.addEventListener('controllerchange', launch);
    }

  } catch (error) {
    console.error('Service worker registration failed', error);
  }
})();

And then let’s create the Service Worker (src/sw.js):

//this is needed to activate the worker immediately without reload
//@see https://developers.google.com/web/fundamentals/primers/service-workers/lifecycle#clientsclaim
self.addEventListener('activate', event => event.waitUntil(clients.claim()));

const globalMap = {
    'react': 'React',
    'react-dom': 'ReactDOM'
};

const getGlobalByUrl = (url) => Object.keys(globalMap).reduce((res, key) => {
    if (res) return res;
    if (matchUrl(url, key)) return globalMap[key];
    return res;
}, null);

const matchUrl = (url, key) => url.includes(`/${key}/`);

self.addEventListener('fetch', (event) => {

  const {request: {url}} = event;

  console.log('Req', url);

  const fileName = url.split('/').pop();
  const ext = fileName.includes('.') ? url.split('.').pop() : '';

  if (!ext && !url.endsWith('/')) {
    url = url + '.jsx';
  }

  if (globalMap && Object.keys(globalMap).some(key => matchUrl(url, key))) {
    event.respondWith(
      fetch(url)
        .then(response => response.text())
        .then(body => new Response(`
          const head = document.getElementsByTagName('head')[0];
          const script = document.createElement('script');
          script.setAttribute('type', 'text/javascript');
          script.appendChild(document.createTextNode(
            ${JSON.stringify(body)}
          ));
          head.appendChild(script);
          export default window.${getGlobalByUrl(url)};
        `, {
          headers: new Headers({
            'Content-Type': 'application/javascript'
          })
        })
      )
    )
  } else if (url.endsWith('.js')) { // rewrite for import('./Panel') with no extension
    event.respondWith(
      fetch(url)
        .then(response => response.text())
        .then(body => new Response(
          body,
          {
            headers: new Headers({
              'Content-Type': 'application/javascript'
            })
        })
      )
    )
  }

});

Here’s what we’ve done here:

We have created the export map, which associates package id with global var name
We have created a script tag in head with contents of UMD-packaged script
We’ve exported the mapped global var as default export of module

For the sake of tech demo this method of patching should be enough, but it may break with other UMD declaration. Something more robust can be used to process sources.

Now let’s adjust the src/index-dev.html to use the bootstrap entry point:

Now we’re able to import React and React DOM.

Static way

It’s worth to mention, that there’s also another way. We can install ESM distributable:

npm install esm-react --save

And then use following map:

{
  "imports": {
    "react": "../node_modules/esm-react/src/react.js",
    "react-dom": "../node_modules/esm-react/src/react-dom.js"
  }
}

But unfortunately this project is quite stale, latest is 16.8.3 whereas React is 16.10.2.

JSX

There are two ways to do the JSX compilation. We can either go traditional way and use Babel to pre-compile or we can use it in runtime. Of course for production it would make much more sense to pre-compile, development mode can be more brutal. Since we already use Service Worker let’s enhance it.

Let’s install a special Babel package that can do it:

$ npm install @babel/standalone --save-dev

Now let’s add following to the Service Worker (src/sw.js):

# src/sw.js
// at the very top of the file
importScripts('../node_modules/@babel/standalone/babel.js');

// activation stuff as before

self.addEventListener('fetch', (event) => {

  // whatever we had before

  } else if (url.endsWith('.jsx')) {
    event.respondWith(
      fetch(url)
        .then(response => response.text())
        .then(body => new Response(
          //TODO Cache
          Babel.transform(body, {
            presets: [
              'react',
            ],
            plugins: [
              'syntax-dynamic-import'
            ],
              sourceMaps: true
            }).code,
            { 
              headers: new Headers({
                'Content-Type': 'application/javascript'
              })
            })
        )
    )
  }

});

Here we’ve used same approach to intercept the network request and respond with slightly different content, in this case we use Babel to transform the original response. Please note that plugin for dynamic import has a different name syntax-dynamic-import, not a usual @babel/plugin-syntax-dynamic-import due to Standalone usage.

CSS

In the above mentioned article author used text transformation, here we will go a bit further and inject the CSS in the page. For that we again will use the Service Worker (src/sw.js):

// same as before

self.addEventListener('fetch', (event) => {

  // whatever we had before + Babel stuff

  } else if (url.endsWith('.css')) {
    event.respondWith(
      fetch(url)
        .then(response => response.text())
        .then(body => new Response(
          //TODO We don't track instances
          //so 2x import will result in 2x