I implemented search and filtering for entities on our product at epilot. The users were heavily using the feature, however, a unique issue surfaced: difficulties in filtering file names containing diacritical marks like umlauts (Äpfel, über, schön, etc.).

Intrigued, I delved into the logic to investigate. Initially, everything seemed in order. For instance, a file named “blöb” was saved precisely as “blöb” — no bizarre encoding alterations were apparent.

Can you spot any difference between “blöb” and “blöb”?

This time instead of typing the word in the search bar, I copied it from the uploaded and saved filename, and and voilà, the search functioned perfectly! 🤯. This definitely meant the file name was subtly altered upon upload, hinting at a background encoding transformation.

So, as anyone would, I employed the power of encodeURIComponent to see for myself what really was different.

There it was! The different types of encoding forms that led to this. Now you might be wondering (I was wondering too!):

As if worrying about encoding wasn’t enough, now we’ve got to stress over its different types too 🤦‍♀ and wait, its 2024, and we are still grappling with Unicode character encoding problems?

The culprits in this case were these two contrasting encoding types: %C3%BC and %CC%88, leading to the search malfunction. This is a well-known issue with NFD/NFC encoding forms.

There is a detailed explanation here.

Now imagine explaining this technical jargon that to a user! 😃

To fix it, I aligned the encoding type of the searched filename with that of the saved filename.
On MacOS, the encoding was NFD (for ö: 111 + 776), whereas on Linux, it switched to NFC (ö: 246).

The solution?

• Post-upload, before saving the file as an entity, I applied .normalize() to the filename, ensuring it was uniformly searchable.
• Additionally, as a cleanup step — to write a migration script to normalize the names of all existing files with non-ASCII characters.

A simple string normalise proved effective in this case. ✅

Read more about String.prototype.normalize() here. There is also specific normalization option to pass to ensure canonically equivalent normalization — for example: string.normalize("NFC")

Here’s a visual representation of the problem with the solution output:

files umlaut.mov

Hope it helps!

When working on the web, we all end up coming across situations where the experience for the user seems a bit janky. This happens either when you are dogfooding, or the users actually highlight facing that slowness, or you see that for yourself in your performance metrics or tools.

What one’d do in such a situation is look at the obvious things first: are the images loaded lazily? Is the bundle size too big? Is the main thread blocked? How is the performance panel recording looking? Is the data coming from the backend taking more time than it should?

If its one of the first problems, we’d fire up our favourite devtools and try to find out the root cause. If its the data being slow, we could see how long that is taking in the network requests duration. And, if it does end up being slow from the backend, the next step would be to dive into some logs and checking what piece of code in the request endpoint takes the time, or embedding some performance information in the response, or using other performance tools etc.

We had a similar situation at epilot where we use Elasticsearch to look for all related products of an order for example, and also all the other entities the order itself might be a relation to.
This meant, looking for direct relations of the order but also the reverse relations of that order.

Now in the Elasticsearch world, this meant:

1. Getting the list of related products of the order.
2. Searching through the documents in elasticsearch for all those related IDs.
3. Also, searching all other docs for having this one order as a relation to this (the reverse relation example, I shared above).
4. Finally returning all the found relations.

This worked great for the first few years and got us exactly what we needed quickly. However, as we scaled, the amount of documents indexed in ES increased too, and that meant, the step 3 above started getting expensive and slow! The logic was to look for an entity ID in all the indexed documents and find if it is a relation or not. For orders with a lot of products, the endpoint would respond in seconds.

We had to fix the logic/ update the design asap.

The first solution was to update the ES query to make it more specific about what is being looked up, and then check if it improved with that query change and how the results looked for the entities with many relations. This meant looking into a crazy amount of Cloudwatch logs and looking at the total time taken and if it was any better.
We made a change to search query to be an IDs specific query which would look through only the IDs in the indexed docs, and not through all the fields in those ES documents.

GET /_search
{
  "query": {
    "ids" : {
      "values" : ["1", "4", "100"]
    }
  }
}

The expectation was this would completely solve the problem and be super quick. This wasn’t the case unfortunately.
What we were missing was to see which part of the code was really adding to the time taken by the endpoint. We needed to observe the different steps in the process and see all the sections where the time taken could be reduced. We needed to see if it was really ES that was the culprit.

This is where Server-Timing API came into picture. Something that would break down the request into pieces and give us an idea about what’s expensive. The API allowed to pass the timings specific to a request from the backend to the browser. This meant that we could now see in the network requests, how the response times looked like, which part of the function took more time and so on.

To support this, we need to send a Server-Timing response header from the backend and time the methods that we needed. Something like this:

HTTP/1.1 200 OK

Server-Timing: db;dur=53, app;dur=47.2

The Server-Timing header can send three bits of information -
- metric name
- duration
- description of the metric

In the above example:
db — metric name
dur=53 — the time in milliseconds thedb metric took (This is for instance, the time to fetch some data from the Database)
app — another metric name and dur=47.2 being the time taken for the same.
The header can take multiple metrics separated by commas delivering great information and is super lightweight. Although it is recommended to keep the metric names as small as possible.

An example of what the returned Server-Timing header looks like:

db; dur=142.715967; desc="getRelationsForEntity",flt; dur=36.777609; desc="filterEntityListByAccess",pgn; dur=22.96549; desc="getPaginatedRelations",hyd; dur=0.64605; desc="hydrateRelations",ddp; dur=0.13311699999999999; desc="dedupeRelations",total; dur=721.388338

And this information then translates into a more visual and helpful view as a part of the network timings tab of the request OR available via the PerformanceServerTiming interface.

To access the three available properties, they correspond in the PerformanceServerTiming as:

• “name” -> PerformanceServerTiming.name
• “dur” -> PerformanceServerTiming.duration
• “desc” -> PerformanceServerTiming.description

Here’s the working draft with more information.

These are not available as a series of timings as such and just simple metrics and therefore, does not show in a waterfall pattern.

This means we can now tackle both the frontend performance problems and the backend slowness problems all in one place. No jumping to the logs to find a response was slow or heading backing again to the devtools to find the frontend problem.

We could also read the data now available using the navigation (PerformanceNavigationTiming) and resource timing (PerformanceResourceTiming) APIs and send it our analytic tools to create metrics/monitors.

const performanceObserver = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    log('Server Timing', entry.serverTiming);
  }
});

performanceObserver.observe({type: 'navigation', buffered: true});

Below is one of the examples from a method taking different amounts of time at different steps in the request. We will set up together the whole setting up and measuring the server response times later in this article.

This was especially helpful for the Elasticsearch example that I shared earlier as we needed to know, which change was improving what part of the implementation and if it was moving us closer to improved response times. The Server-Timing header does not directly improve the timings but helps find out what could be worked towards.

The uniqueness of the API is that its super flexible and allows the server to communicate important information to the browser, even other than the server timings. It would be amazing to have this supported on Safari as well, which currently lacks the API support.

Coming back to how we came down to the API response in 400ms seconds as compared to 1.8s earlier

These are the two versions of the API. When we measured the performance with server-timings on the ES implementation after switching to the IDs search query on Elasticsearch, the performance seemed to be taking all the time especially on this function searchRelationsForEntity .

Now we knew this was to do with ES but where exactly. We timed specifics inside this method.
On measuring further, it was entitySearch.searchEntities that was taking up the most time so we needed to find a solution that would not go through all the ES documents at all to get the relation entities.

This is where changing the design made sense, and we moved to a graph table design with Dynamo, which would store all the relationships as mappings, and return all the entity IDs whether as direct relations or reverse.
Now we could choose if we wanted to search ES with all the IDs that we have or just batch get the entities from the db, and support pagination (this was an added advantage with ES) etc. separately.

Now, this complete dynamo approach saved us looking through all the indexed documents for the IDs that we need, and also cut some slack on Elasticsearch which would have this unnecessary load, eventually affecting other straightforward searches as well.

This is how the server timings looked with the DynamoDB approach:

This was an amazing improvement, clearly visible with this significant performance information right in the browser devtools for us. This also ended up helping improve the code in general, to gain some more milliseconds by refactoring, and using performant ways to filter/hydrate/paginate things etc.

We love to see the output of what’s in the works, and this was a great example to look at and feel a sense of achievement for.

Implementation of Server-Timing as a middleware for AWS Lambda 💪

Now if you look up the existence of libraries supporting server-timing header, there are so many but most of them for Express and NodeJS.

• server-timing
• server-timings etc.

We needed to create one for the serverless use case which could be used for AWS Lambda, lets say to use with middy for example.

Step 1. Create a withServerTimings() middleware to use for AWS lambda

import middy from '@middy/core';
import * as Lambda from 'aws-lambda';

export const withServerTimings = () => {
    return {
      // add Server-Timing header to response
      after: (handler: middy.Request) => {
        const response = handler.response as Lambda.APIGatewayProxyStructuredResultV2;

        try {
          const headers: unknown[] = [];
          /** 
              This is where the headers with different metrics are be saved 
              and retrieved from
           */
          const timings = getServerTimingHeader(headers); 

          response.headers = {
            ...response.headers,
            'Server-Timing': timings,
          };
        } catch (e) {
          Log.debug(`Middleware Error: Could not record server timings`, e);
        }
      },
    };
};

This creates a usable withServerTimings() middleware to then be used as:

Step 2. Apply the middleware to the request handler

return middy(handler)
    .use(...)
    .use(withCors())
    .use(withServerTimings())
    .use(...)

This already ensures that your backend will now be sending a Server-Timing header in the request endpoints.

Next step would be to actually time our operations and store them as a common timing variable which can then be sent to the getServerTimingHeader method for example.

Step 3. Implement timing for your methods

const times = new Map<string, Record<string, unknown>>();

// starting the timer on a method
const startTimer = (name: string, description?: string) => {
  times.set(name, {
    name,
    description: description || '',
    startTime: process.hrtime(),
  });
};

// ending the timer and recording the duration of the method
const endTimer = (name: string, description?: string) => {
  const timeObj = times.get(name);
  if (!timeObj) {
    return console.warn(`No such name ${name}`);
  }
  const duration = process.hrtime(timeObj.startTime as [number, number]);
  const value = duration[0] * 1e3 + duration[1] * 1e-6;
  timeObj.value = value;
  times.delete(name);

  return timeObj;
};

// finally setting the metric value received from endTimer object

const metric =
    typeof description !== 'string' || !description
      ? `${name}; dur=${dur ?? 0}`
      : `${name}; dur=${dur}; desc="${description}"`;

This would then be used as:

Step 4. Usage of timers of the methods

startTime('db', 'getRelationsForEntity');
  const relations = await getRelationsForEntity({
    ...
  });
  endTime('db', 'getRelationsForEntity');

  .
  .
  .

  startTime('pgn', 'getPaginatedRelations');
  const { relations: paginatedRelations, hits } = getPaginatedRelations({
    ...
  });
  endTime('pgn', 'getPaginatedRelations');

And there we go, we will have the timed methods and the server headers looking like:

db; dur=139.99; desc="getRelationsForEntity",flt; dur=36.68; desc="filterEntityListByAccess",
pgn; dur=39.37; desc="getPaginatedRelations",hyd; dur=0.61; 
desc="hydrateRelations",ddp; dur=0.130; desc="dedupeRelations",
total; dur=741.53

All the timing implementation and setting up the middleware can be a hassle, and should be a one-time job. I created a package out of this implementation called lambda-server-timing which we can simply use by importing the things we need, i.e.,

• lambda-server-timing
• GitHub - NishuGoel/lambda-server-timing: Enables Lambdas to return responses with Server-Timing Header allowing to to pass request-specific timings from the backend to the browser.

• withServerTimings to use a middleware
• startTime to start measuring a function/set of functions
• endTime to end the timing of the function/set of functions and get the metric set.

import {withServerTimings, startTime, endTime } from 'lambda-server-timing';

Lets time our functions down! 💪

References

• Working draft
• MDN Server Timing
• Navigation Timing
• Resource Timing

This post explains how to setup a new elasticsearch cluster from scratch and make it ready to be used for indexing data into, communicating to/from as a remote cluster, along with setting up monitoring for the new cluster.

Problem 1: Create a new elasticsearch cluster on elastic.co based on the required configuration for your use case.

To create a new cluster from scratch:
Create a main.tf file with following resources:

a. ec_deployment - The main cluster deployment resource
b. aws_secretsmanager_secret- The cluster secrets
c. aws_secretsmanager_secret_version - The cluster secret version needed for the secrets
d. elasticstack_elasticsearch_cluster_settings - To add default settings to the cluster like cluster nodes etc.

Example: ec_deployment

resource "ec_deployment" "<cluster-name>" {
  name = "<cluster-name>"

  region                 = "aws-eu-central-1"
  version                = var.elasticsearch_version
  deployment_template_id = standard/arm-based (decide based on the requirement)

  elasticsearch {
    autoscale = "false"
    topology {
      id   = "hot_content"
      size = "15g" // size of the cluster
      zone_count = 3 // number of nodes 
    }
  }
  kibana {}

  // important to include this in the cluster config to ensure to NOT override any existing remote cluster related config
  # ignore changes to remote cluster config
  lifecycle{
    ignore_changes = [
      elasticsearch[0].remote_cluster
    ]
  }
}

Few important bits to note:
Ensure the region and ES version are consistent for your cluster.

By default, deployment_template_id should be defined as standard aws-storage-optimized-v3 but if your are creating the cluster for memory-intensive work, consider creating it as arm-based aws-general-purpose-arm-v6

The size and number of nodes vary based on the need, so decide it accordingly from the Elastic Cloud UI.

An important bit to always add in your cluster deployment is to ignore changes to the remote cluster config. This ensures the remote communication that is configured among clusters doesn’t get overriden by this update.

ignore_changes = [
      elasticsearch[0].remote_cluster
    ]

Example: Cluster Secrets

# Create the AWS Secret Manager secret
resource "aws_secretsmanager_secret" "<cluster-secret-name>" {
  name = "/elasticsearch/<cluster-name>/credentials"
  description = "Elasticsearch credentials for <cluster-name> cluster"
}

# Create the AWS Secret Manager secret version with JSON value
resource "aws_secretsmanager_secret_version" "<cluster-secret-name>" {
  secret_id     = aws_secretsmanager_secret.<cluster-secret-name>.id
  secret_string = <<JSON
{
  "endpoint": "${ec_deployment.<cluster-name>.elasticsearch.0.https_endpoint}",
  "username": "${ec_deployment.<cluster-name>.elasticsearch_username}",
  "password": "${ec_deployment.<cluster-name>.elasticsearch_password}"
}
JSON
}

Example: cluster_settings

resource "elasticstack_elasticsearch_cluster_settings" "<cluster-settings>" {
  elasticsearch_connection {
    username = ec_deployment.<cluster-name>.elasticsearch_username
    password = ec_deployment.<cluster-name>.elasticsearch_password
    endpoints = [ec_deployment.<cluster-name>.elasticsearch.0.https_endpoint]
  }

  persistent {
    setting {
      name  = "cluster.max_shards_per_node"
      value = "2500"
    }
    setting {
      name  = "cluster.routing.allocation.cluster_concurrent_rebalance"
      value = "30"
    }
    setting {
      name  = "cluster.routing.allocation.node_concurrent_recoveries"
      value = "30"
    }
  }
}

Three important settings that would make sense to add as default 👆

• max_shards_per_node the value would vary based on the cluster size and requirement
• cluster_concurrent_rebalance To allow correct rebalancing of shards to all available nodes (20-30 is a good value for this)
• node_concurrent_recoveries To allow correct rebalancing of shards to all available nodes (20-30 is a good value for this)

Above steps will create a cluster with the defined config. ✅

Always good to ensure and check what our changes are adding/updating/deleting, to be extra cautious to NOT end up deleting/updating some existing resource. 😨

Problem 2: Add the newly created elastic cluster to the POOL of clusters in the DynamoDB Cluster assignment table

Now that the new cluster is and available to view on elasticsearch as a healthy deployment, we can make the next move to include it in the list of available clusters, to be able to then use it for our ES needs.

example:

To add it to the pool, you can assign it to the table where cluster assignment is defined, in our case: ClusterAssignmentTable

To add your new cluster to this pool, use:

json=$(cat <<EOF
{
  "pk": {"S": "POOL"},
  "sk": {"S": "CLUSTER#${cluster_name}"},
  "cluster": {"S": "${cluster_name}"},
  "endpoint": {"S": "${endpoint}"},
  "username": {"S": "${username}"},
  "password": {"S": "${password}"}
}
EOF
)

echo "===> Storing cluster $cluster_name in table $table_name..."
aws dynamodb put-item \
    --table-name $table_name \
    --item "${json}"

As a result of this, the new cluster should be found in the cluster assignment table.

Problem 3: Allow it the access to communicate to/from other existing ES clusters.
When dealing with multiple ES clusters, there is the requirement to query data from one cluster to another. In the current implementation, we already have use cases of doing multi-org queries where those orgs could belong to different clusters.

Usage looks like:

To allow this, we need to ensure, every time a new cluster is created, its also set up for the remote cluster communication.

To do this, you could do something like for all your clusters in the pool:

payload=$(cat <<EOF
{
  "persistent": {
    "cluster": {
      "remote": {
        "${remote_cluster}": {
          "mode":"proxy",
          "server_name": "${remote_server_name}",
          "proxy_address": "${remote_proxy_address}",
          "skip_unavailable": true
        }
      }
    }
  }
}
EOF
)

curl -sS -X PUT "$endpoint/_cluster/settings" \
  -u "$username:$password" \
  -H 'Content-Type: application/json' \
  -d "$payload" | jq

This will ensure all the clusters for that environment are setup for remote communication to one another.

Also, remember we had that extra step to ignore changes to the remote cluster config? This is why! We DO NOT want to override any remote config changes when updating clusters.

At this point, you have your cluster ready on elasticsearch, cluster allocation table, and also able to talk to other clusters 💪

And then there are the Not-so-good times when clusters are not doing that great and healthy.

With these many clusters (even more probably in the future), we want to know the whereabouts of each of them, to be better at managing these.
So a cluster specific monitoring is a good idea. We do that using Datadog. Datadog provides a Elastic Cloud integration that can be used for the same.

Something like above 👆

Hope this helps!

Spin up a new Elasticsearch cluster? was originally published in epilot on Medium, where people are continuing the conversation by highlighting and responding to this story.

It happens to be a series of one type of things that I do, and I do a lot of that for the year. For example, 2018 was “writing blog posts” year, wrote more than 50 blog posts. 2019, “the reading books” years, read 36 or so books that year. 2021, again a lot of writing that year, 2022, “insane traveling” year.

TL;DR

When I started this year, I was aware of one travel that was going to happen, for sure. It was the one that involved me moving to a different country to work. However, there was so much more in store than what I expected.
It felt like telling Covid: Enough of your shit!
The point is, I ended up traveling to all the countries that I would think about traveling to, one day. And I wouldn’t have been writing this proudly, if it was not work-related. All of the travel was tech related. And what can be better than that?

Started the year with speaking at my favourite conference — ngIndia, and the energy was crazy since it was happening now after covid. Knowledgeable speakers, getting to know people.

I joined epilot beginning of this year which involved lots of learning and working with amazing people and in the best work culture one could have.
One important thing I realised was:

“A company is really doing something right, if even its engineers talk about the product (bigger picture) with as wide eyes and great passion as the founders.”

And I see this at epilot everyday and this inspires me so much to have such a culture at my own company, someday.
Like I mentioned in the beginning, this year was so much traveling and I am going to write small interesting instances from all my trips here, so I can come back later to this and smile.

Started with me traveling to the company summit in Germany which involved a week of fun with colleagues. This was when I got to know that a beer CAN cost less than water sometimes if you are in Germany :) Also saw, the team that parties crazy also achieves crazy.

And before it starts sounding like an epilot appreciation post, I should also mention the amazing time I had at the GDE summit in Leh Ladakh. All the Google developer experts coming together after covid, meeting in person, doing activities, playing games, was exactly what was needed after those two dreadful years.

September brought the wave of travel and never stopped until December. I moved to a new place here in Köln, and the next day I was traveling to Romania already for a web talk at Devfest Craiova about JavaScript Surprises. It was a one day event meeting amazing developers followed by a wonderful day out in the city.

Visited the Nicolae Romanescu Park, the largest natural park in Eastern Europe, apparently. I think the highlight of the Romania trip would be the Papanași that I had and the Sârbă dance we did at a get together dinner event.

This was followed by a surprise opportunity to visit the Firebase Summit in New York as an Angular GDE. And New York is a dreamy city until you have visited it at least once, and after that one decides for himself, if NYC is still as exciting for them. To each their own.
This was great fun as well with fellow GDEs exploring all the touristy stuff — Times Square, Wall street, Helicopter View of the city, Brooklyn bridge, Statue of Liberty, Cruise tour, Summit One Vanderbilt etc. The highlight has to be the event venue which surrounded The Little Island.

Next was a talk at JavaScript Poland in Warsaw, which was a trip full of learning. Attending workshop on Qwik, learning about Partytown, Solid.js and reactivity conversation with Ryan, Streaming concept and Out of Order streaming, etc. The highlight would be meeting Dariusz after 3 years of meeting him for the first time in ngIndia and the beautiful JS Poland stage.

Had a work vacation planned in Vienna, Austria in November and boy, the city is beautiful. We stayed at a hostel called Jo and Joe which had a creative element to it. We walked around the city and I got to see the first snow of the season there. The highlight of the trip was really just enjoying, playing with colleagues, winning the Dart game, but cluelessly :)

Last but not the least was the trip to London for another Devfest event. This was quite unexpected as I had visa issues until two days before the travel, but ended up working out a day before, and how could I miss it then. Shared about JavaScript Memory Usage at Devfest London, again followed by a day out in the city, again some touristy stuff, but also great conversation with fellow speakers.

I happened to be in London during Christmas and that in itself is the best highlight, but there were so many other. It snowed there making it even more beautiful, sitting in those London special red buses. Walking around the city and having conversation with a friend was what made it all worth.

You know how when we realise a pattern, it is no more fun to still let it going. We, at least I, like to see if the pattern can be broken and can something good come out of it. I think I would really not want this year to be a series of one kind of thing.
Would really like it to be a mix of all kinds of things I like to, need to do, with the right amount of balance, consistency, etc. Looking forward to writing, reading, creating more value this year. :)

Also, moving is quite an adventure. We introduce new music, food, culture, and new habits in general to our lives, sometimes language as well. Example being, :D

Some talks:
- Devfest London
- JS Poland
- Devfest Craiova (recording)
- Devfest Bangalore
- ngIndia (recording)
- Web Directions — Global Scope

Some slides:
- microFrontends
- JavaScript Surprises
- Understanding JS memory Usage

Some writings:
- Breaking up long tasks — Perf Calender
- Dabbling at epilot
- Unraveling the Web
- Svelte-i18next

Some books read:
- The midnight library
- The T-shirts I love
- Thinking fast and Slow
- AI 2041
- The Bitcoin Standard
etc.

Prost! 🍻

This is the kind of experience, we as web engineers, cannot afford to give to the users. And there can be multiple reasons that could lead to this. If you regularly check the performance of your apps, you would have come across suggestions to code-split your application and reduce shipped JavaScript. In this post, we look at the causes of slow responsiveness of an application, how we could measure the slowness, and the approaches to improve it.

TL;DR
Certain tasks on a user interaction can be blocking, giving the user a broken experience, and resulting in a low INP metric score.

Some ways to optimise it could be:

• keep tasks small and dedicated to what they should do.
• use Chrome’s scheduler API to schedule tasks for browser
• yield important tasks to the main thread

Let us first understand why a user interaction could be slow. When a user interacts with an element on the browser, the user agent takes that as one task and queues it in the list of tasks for the browser. This one task could take more time or less depending on the type of operation needed to be performed. Some tasks take longer resulting in the main thread being busy performing that and leading to the the other tasks being blocked in the meantime.

Long tasks

A task is considered a long one if it goes beyond 50ms to execute and 50ms to respond, basically, more than 100ms. A website which doesn’t have long tasks, is expected to respond within 100ms with both the execution and response time on a user input, according to RAIL.

As per data from Web Almanac, it’s interesting to note that a median mobile page tends to have 19 long tasks.

This is where INP comes into picture. INP is a responsiveness metric, stands for Interaction to next paint, which measures the time from when the user interacted with the page until when the next frame was actually visible on the page. A good INP measure is when the time between interaction and the feedback as a result of that interaction is the least.
An interaction could mean events like tap, click, keyUp, pointerdown etc.

An interesting read here on how’s INP actually measured

.

INP is calculated by observing all the interactions made with a page. The chosen value is then a percentile of those interactions. A formula is then used to choose a high percentile value of those interactions.

The way we put web vital measures in a Good, Needs Improvement, Bad buckets, here is what a good and poor looks like for the INP metric.

To calculate the overall page responsiveness, an INP measure is calculated when the user navigates away from a page, and this comprises of all interactions like mouse events, keyboard events, touch/tap on a touchscreen device.

FID / INP?

I remember analysing my websites, and always being satisfied with at least one metric of the three core web vitals 😀 — the First Input Delay. Would up my ego a bit that I am doing something good for sure. But if we look at the bigger picture, it is noticed that most websites have a good FID value, even when the responsiveness experience isn’t that great.
And to address this, INP with a better measurement approach has been introduced. INP doesn’t just measure the first interaction but all interactions on a page, unlike FID. Also, as the name says, INP measures the time until the next paint happens, unlike FID, that measures the time only until the processing happens.

If we look at the INP scores on mobile devices, 55% of websites have a good INP score, and 36% need improvement. However, in case of FID, 92% of websites having “good” FID. So, a better responsiveness measure with INP already brought down the “good” score.

It is also interesting to see that INP score tends to be inversely proportional to the popularity of a website, as per data from the Performance chapter in web almanac.

With a better approach to measure the responsiveness of a website with INP, it would make sense for developers to really start considering this metric as an important performance check, also with Google considering to replace FID with INP as one of the core web vitals.

When looking at the mobile experiences by site rank, 52% of top 1K websites fall in the good category with FID metric, however only 20% pass with good status with INP as a metric, and this is a huge difference hinting towards the need to really improve the overall responsiveness situation, long tasks being one of the things to solve.

Can I track my website’s INP value?

One cool way to track our website’s interaction to next paint measure with real data from users is using the web-vitals library.

The library offers the onINP method that returns the type of event along with the INP value.

We could record this data and then receive it via some API or analytics provider to be able to understand the reasons behind the INP measure based on the real usage from field.

Something like this:

import {onINP} from 'web-vitals';
onINP(({name, value, attribution}) => {
  console.log(name)
  console.log(value)
  console.log(attribution.eventType)
});

This gives us the measure of our INP and helps us understand further how we this interaction delay be improved.

This is how the general INP score looked like on websites as per the data from Web almanac.

How do you find a long task in your application?

By now, we would know how the INP value is doing, and the next step right away would be to profile our app and see the improvement areas. To do this, we go straight to the performance profiler and identify the long tasks (ones exceeding the threshold of 50ms).

As we can see, there are many long tasks leading to slower input response to the user, one such having a delay of 676ms.

And as we dive deeper, it gives us some idea about the operations causing this to go up to 600ms here:

Let us try to optimise one such long task now. There can be different approaches to improve long tasks, with one being breaking up your long tasks into smaller dedicated tasks.

For example, the long task from above which took 600ms, was a blocking task for UI related updates, and thus would result in a broken experience for the users since they’d see a delayed response. What if the task still does exactly what it does but it is broken into multiple tasks with small operations, meaning:

• Task 1 — Only for querySelector step
• Task 2 — parse
• Task 3 — getOperations
• Task 4 — removeChild
  and so on.

Read more here 👇🏻

Breaking up long tasks

Blessed to have been surrounded by the best people in the field. Dhananjay, who introduced me to the JavaScript community, himself a 12-times MVP. Deepak Rajendran who is a constant motivation to share more of what we do. The rest of the community who gives the opportunities to learn and grow together with others.

Here is my MVP profile.

Hope to continue to keep learning, sharing, growing. Listing some of my contributions from the past year:

• OSS — Created a svelte wrapper for using i18next
• Speaker — Microfrontends at ngIndia 2022
• Blog — Work/Life update
• Meetup Session — How the web/JavaScript has evolved over the years
• GDE Community Highlight
• Blog — How to: Svelte i18next
• Author — JavaScript 2021 Web Almanac
• Podcast (Spotify, Apple, Google) — Unraveling the Web with me

Year 2022, this was after more than 2 years that we got to be at an in-person conference, and gosh, the experience! I am forced to accept that we were simply compromising when being a part of the online conferences.

I think conferences were anyway never about only consuming the content delivered from the sessions. It was majorly about the networking, the warmth from meeting people. So when we went to ngIndia 2022 event this May, it was about meeting those people from online bonds from the past two years and connecting NOT from behind the screens.

So, 21st May, I got the opportunity to speak at #ngIndia on the topic ‘Microfrontends’. Now I chose this topic because I am currently working with micro-frontends at @epilotGmbH, and what could be better than talking about the success and challenges of using micro-frontends for large-scale apps than this.

So my preparation did include solving issues related to, for example, cross-micro-frontend communication AT WORK and coming back home and trying to explain the problem and its solution from the day through my slides.

Finally I decided to do my talk in the form of a role-play with Srashti Jain to help audience understand better.

Me — I wish I could work for this company but I haven’t used X framework for a while now.

Srashti Jain (as a recruiter) — oh that’s not a problem at all! Our product is based on a micro-frontend architecture. We’d love to hire you to work on your choice of framework and build our awesome product.

Me — Wow that sounds interesting! So I don’t need to skill up in another framework (X framework) now to contribute to this product. Awesome!

So this is how I began my talk to explain how micro-frontends made scaling up teams and projects framework-agnostic. But thats not it, I am sharing my slides here to share a sense of what was discussed, however, we will soon have the recorded talk up live and ready to be shared here.

My experience speaking about micro-frontends was beyond my expectations, and I had such important conversations with many engineers from the audience. Our discussions revolved around how developers could use single-spa framework to deal with their micro-frontend architecture as the top level router, around how developers faced routing issues in their current single-spa MFEs, how some developers desired to switch to micro-frontends to migrate from legacy codebases like that of AngularJS etc.

The conference had more than 500 people joining in the fun, with the right element of over-coffee networking, over-lunch chit-chat, after-talk questions.

Speakers from all over the world shared interesting findings during their talks. The conference was followed by an after party where the speakers and the volunteers partied together with food, drinks, music, dance and what not!
During the pre-conference dinner, the speakers were felicitated with gifts and swags from ngIndia organiser, Dhananjay Kumar.

I always love to speak at ngIndia and looking at the love from people who attend this event yearly, I am proud to be a home grown speaker of ngIndia, saying this since this is where my foray into speaking at events to share my learning started.

Here are some of the captured moments from the conference.

Thank you!!!

Stay tuned for the talk recordings :)

https://www.ng-ind.com/

Matching faces to Twitter DPs #ngIndia was originally published in epilot on Medium, where people are continuing the conversation by highlighting and responding to this story.

Presenting the svelte wrapper for i18next.

This library, based on i18next, wraps an i18next instance inside a svelte store and observes the i18next events like languageChanged, resource added, etc. to be able to render our svelte components.

Package:-

svelte-i18next

GitHub:-

GitHub - NishuGoel/svelte-i18next: Internationalization for svelte framework. Based on i18next ecosystem

The implementation looks like this:

The package creates a Svelte writable of i18next and listens to the events triggered by any updates on the i18next instance.
For example, if an application loads a new namespace for their translations, the svelte_i18next package will trigger the
i18next.on(‘loaded’, function(loaded) {}) event.

Check the i18next events here.

Usage:

i18n.js

import i18next from "i18next";
import { createI18nStore } from "svelte-i18next";

i18next.init({
   lng: 'en',
   resources: {
     en: {
       translation: {
         "key": "hello world"
       }
     }
   }
});

export const i18n = createI18nStore(i18next);

App.svelte

<script>
  import i18n from './i18n.js';
</script>

<div>
    {$i18n.t('key')}}
</div>

See a example usage here: Svelte example

You could also use namespaces as you would using i18next and the svelte wrapper will re-render the translations based on the namespace provided.

import { createI18nStore } from 'svelte-i18next'

const i18n = createI18nStore(i18n_instance) 
i18n.loadNamespaces(['example']) // this triggers the re-render and loads translations from the provided namespace.

<div>
    {$i18n.t(key)}
</div>

Into svelte? Dealing with localisation?

Show some love @svelte-i18next ❤

svelte-i18next was originally published in epilot on Medium, where people are continuing the conversation by highlighting and responding to this story.

Year 2022 marked an amazing start for me with a new stint with epilot GmbH based out of Cologne, Germany. I joined the team to be working on their micro-frontend architecture, with my primary focus going to be on the performance aspects of the applications.
Like the first recent addition we did after I started was to add brotli compression to each of our micro-frontend (MFEs) bundles.

Let’s look at the tech stack involved.

Micro services

The main epilot app is built using microfrontends with each entity as its own application. There is a login application that takes care of the user login state (built using React + TypeScript), a topbar application that has features like Global search, notifications (React + TypeScript), a sidebar application that navigates the user to different entities based on user permissions (Svelte + TypeScript + Tailwind), and so on.

The component library of the product is based on material UI, built with React.

@epilot/base-elements

Due to the open source experience of many engineers in the team, most of the non-confidential stuff is kept public on GitHub and npm.
One of the very interesting things I got to do to ramp up was to build a part of the product as a Svelte MFE. With most of the existing MFEs built using react, this was quite interesting and challenging to do.

We use single-spa framework to build our microfrontends and use different frameworks per the ease and knowledge of an engineer. To make the development journey easier for a new developer starting up with microfrontends, we have a mfe-app-generator package that creates a boilerplate application with the required config.

@epilot360/mfe-app-generator

Due to performance reasons, one of the goals was to switch to rollup for bundling all our applications, however, we use webpack due to rollup’s known issues with aws-sdk.

We use serverless microservices architecture exposed via APIs. Most of the backend services are written in TypeScript using AWS lambda functions, APIs, Cloudfront distribution etc. More on that here.

With Svelte added to our microfrontend stack, we plan to further diversify the frameworks to be used, ensuring to not lose focus on the performance aspect of the application. Here’s a list of support frameworks by single-spa.

I also plan to share our journey from a high LCP (largest contentful paint) to a much reduced one, but I’d keep it for another blog post.

When working with custom implementation and bundling, one of the challenging tasks is to configure your bundler and ensure it is consistent throughout all the applications (MFEs in this case). We have defined a @epilot360/webpack-config-epilot360 to base one’s webpack config on top of the existing basic config defined.

@epilot360/webpack-config-epilot360

I will share more on the webpack configuration for microfrontends and cache invalidation strategies in another post but cache invalidation is definitely my major ongoing focus area.

Epilot SDK

All public epilot services are available through an epilot-sdk (written in JS/TS) that allows developers to interact with epilot APIs. Internally as well, the recommended way to use the services or APIs using directly the SDK instead of going the granular way.

For e.g.,

import { getClient } from 'epilot-sdk/entity-client'

Want to scratch your itch? Find it here:

@epilot/epilot-sdk

Communicating between microfrontends

One of the interesting challenges I recently faced was to communicate between two applications without any shared library. One was the sidebar built using Svelte and another the topbar built with React. Now we usually tend to avoid any such communication between our MFEs but this was a state clear event to happen before a routing change in any of the two MFEs.

I used single-spa’s custom events to do this. The idea was to clear the selected state of an icon in the topbar when an icon in the sidebar was selected, and vice versa.
To do this, I used the single-spa:before-routing-event custom event to check if the active URL in the current application was undefined, we could clear the state, if however, the URL existed, that becomes the active state.
Checking the URL history without communicating between two applications was not possible due to two separate routers for each application. Svelte app using the svelte-routing package where with React, the react-router .

Read more about the single spa custom events here.

Here’s a detailed view of the tech stack used at epilot:

https://docs.epilot.io/techradar

I have shared more info on my journey to epilot as a full-stack developer in my Spotify Podcast here:

Interested to know more about how we do things at epilot? Reach out to me here: twitter.com/@TheNishuGoel

dabbling with ____ @epilot-dev was originally published in epilot on Medium, where people are continuing the conversation by highlighting and responding to this story.