In his novel book ‘Atomic Habits’ the author James Clear states that:

“Mastery is created by habits. However, sometimes when we’re on auto-pilot performing habits, we tend to slip up… Just being we are gaining experience through performing the habits does not mean that we are improving. We actually go backwards on the improvement scale with most habits that turn into auto-pilot”. In other words, practice makes perfect, and bad practices make things worst

We copy-paste mentally and physically things that we are used to, but these things are not necessarily right anymore. Like animals who shed their shells or skin to adapt to a new reality, so the Node.js community should constantly gauge its existing patterns, discuss and change

Luckily, unlike other languages that are more committed to specific design paradigms (Java, Ruby) — Node is a house of many ideas. In this community, I feel safe to question some of our good-old tooling and patterns. The list below contains my personal beliefs, which are brought with reasoning and examples.

Are those disruptive thoughts surely correct? I’m not sure. There is one things I’m sure about though — For Node.js to live longer, we need to encourage critics, focus our loyalty on innovation, and keep the discussion going. The outcome of this discussion is not “don’t use this tool!” but rather becoming familiar with other techniques that, under some circumstances might be a better fit

TOC — Patterns to reconsider​

1. Dotenv
2. Calling a service from a controller
3. Nest.js dependency injection for all classes
4. Passport.js
5. Supertest
6. Fastify utility decoration
7. Logging from a catch clause
8. Morgan logger
9. NODE_ENV

1. Dotenv as your configuration source​

💁‍♂️ What is it about: A super popular technique in which the app configurable values (e.g., DB user name) are stored in a simple text file. Then, when the app loads, the dotenv library sets all the text file values as environment variables so the code can read this

// .env file
USER_SERVICE_URL=https://users.myorg.com

//start.js
require('dotenv').config();
//blog-post-service.js
repository.savePost(post);
//update the user number of posts, read the users service URL from an environment variable
await axios.put(`${process.env.USER_SERVICE_URL}/api/user/${post.userId}/incrementPosts`)

📊 How popular: 21,806,137 downloads/week!

🤔 Why it might be wrong: Dotenv is so easy and intuitive to start with, so one might easily overlook fundamental features: For example, it’s hard to infer the configuration schema and realize the meaning of each key and its typing. Consequently, there is no built-in way to fail fast when a mandatory key is missing — a flow might fail after starting and presenting some side effects (e.g., DB records were already mutated before the failure). In the example above, the blog post will be saved to DB, and only then will the code realize that a mandatory key is missing — This leaves the app hanging in an invalid state. On top of this, in the presence of many keys, it’s impossible to organize them hierarchically. If not enough, it encourages developers to commit this .env file which might contain production values — this happens because there is no clear way to define development defaults. Teams usually work around this by committing .env.example file and then asking whoever pulls code to rename this file manually. If they remember to of course

☀️ Better alternative: Some configuration libraries provide out of the box solution to all of these needs. They encourage a clear schema and the possibility to validate early and fail if needed. See comparison of options here. One of the better alternatives is ‘convict’, down below is the same example, this time with Convict, hopefully it’s better now:

// config.js
export default {
  userService: {
    url: {
      // Hierarchical, documented and strongly typed 👇
      doc: "The URL of the user management service including a trailing slash",
      format: "url",
      default: "http://localhost:4001",
      nullable: false,
      env: "USER_SERVICE_URL",
    },
  },
  //more keys here
};

//start.js
import convict from "convict";
import configSchema from "config";
convict(configSchema);
// Fail fast!
convictConfigurationProvider.validate();
//blog-post.js
repository.savePost(post);
// Will never arrive here if the URL is not set
await axios.put(
  `${convict.get(userService.url)}/api/user/${post.userId}/incrementPosts`
);

2. Calling a ‘fat’ service from the API controller​

💁‍♂️ What is it about: Consider a reader of our code who wishes to understand the entire high-level flow or delve into a very specific part. She first lands on the API controller, where requests start. Unlike what its name implies, this controller layer is just an adapter and kept really thin and straightforward. Great thus far. Then the controller calls a big ‘service’ with thousands of lines of code that represent the entire logic

// user-controller
router.post('/', async (req, res, next) => {
    await userService.add(req.body);
    // Might have here try-catch or error response logic
}

// user-service
exports function add(newUser){
    // Want to understand quickly? Need to understand the entire user service, 1500 loc
    // It uses technical language and reuse narratives of other flows
    this.copyMoreFieldsToUser(newUser)
    const doesExist = this.updateIfAlreadyExists(newUser)
    if(!doesExist){
        addToCache(newUser);
    }
    // 20 more lines that demand navigating to other functions in order to get the intent
}

📊 How popular: It’s hard to pull solid numbers here, I could confidently say that in most of the app that I see, this is the case

🤔 Why it might be wrong: We’re here to tame complexities. One of the useful techniques is deferring a complexity to the later stage possible. In this case though, the reader of the code (hopefully) starts her journey through the tests and the controller — things are simple in these areas. Then, as she lands on the big service — she gets tons of complexity and small details, although she is focused on understanding the overall flow or some specific logic. This is unnecessary complexity

☀️ Better alternative: The controller should call a particular type of service, a use-case , which is responsible for summarizing the flow in a business and simple language. Each flow/feature is described using a use-case, each contains 4–10 lines of code, that tell the story without technical details. It mostly orchestrates other small services, clients, and repositories that hold all the implementation details. With use cases, the reader can grasp the high-level flow easily. She can now choose where she would like to focus. She is now exposed only to necessary complexity. This technique also encourages partitioning the code to the smaller object that the use-case orchestrates. Bonus: By looking at coverage reports, one can tell which features are covered, not just files/functions

This idea by the way is formalized in the ‘clean architecture’ book — I’m not a big fan of ‘fancy’ architectures, but see — it’s worth cherry-picking techniques from every source. You may walk-through our Node.js best practices starter, practica.js, and examine the use-cases code

// add-order-use-case.js
export async function addOrder(newOrder: addOrderDTO) {
  orderValidation.assertOrderIsValid(newOrder);
  const userWhoOrdered = await userServiceClient.getUserWhoOrdered(
    newOrder.userId
  );
  paymentTermsService.assertPaymentTerms(
    newOrder.paymentTermsInDays,
    userWhoOrdered.terms
  );

  const response = await orderRepository.addOrder(newOrder);

  return response;
}

3. Nest.js: Wire everything with dependency injection​

💁‍♂️ What is it about: If you’re doing Nest.js, besides having a powerful framework in your hands, you probably use DI for everything and make every class injectable. Say you have a weather-service that depends upon humidity-service, and there is no requirement to swap the humidity-service with alternative providers. Nevertheless, you inject humidity-service into the weather-service. It becomes part of your development style, “why not” you think — I may need to stub it during testing or replace it in the future

// humidity-service.ts - not customer facing
@Injectable()
export class GoogleHumidityService {
async getHumidity(when: Datetime): Promise<number> {
    // Fetches from some specific cloud service
  }
}
// weather-service.ts - customer facing
import { GoogleHumidityService } from './humidity-service.ts';
export type weatherInfo{
    temperature: number,
    humidity: number
}
export class WeatherService {
  constructor(private humidityService: GoogleHumidityService) {}
  async GetWeather(when: Datetime): Promise<weatherInfo> {
    // Fetch temperature from somewhere and then humidity from GoogleHumidityService
  }
}
// app.module.ts
@Module({
  providers: [GoogleHumidityService, WeatherService],
})
export class AppModule {}

📊 How popular: No numbers here but I could confidently say that in all of the Nest.js app that I’ve seen, this is the case. In the popular ‘nestjs-realworld-example-ap[p’]() all the services are ‘injectable’

🤔 Why it might be wrong: Dependency injection is not a priceless coding style but a pattern you should pull in the right moment, like any other pattern. Why? Because any pattern has a price. What price, you ask? First, encapsulation is violated. Clients of the weather-service are now aware that other providers are being used internally. Some clients may get tempted to override providers also it’s not under their responsibility. Second, it’s another layer of complexity to learn, maintain, and one more way to shoot yourself in the legs. StackOverflow owes some of its revenues to Nest.js DI — plenty of discussions try to solve this puzzle (e.g. did you know that in case of circular dependencies the order of imports matters?). Third, there is the performance thing — Nest.js, for example struggled to provide a decent start time for serverless environments and had to introduce lazy loaded modules. Don’t get me wrong, in some cases, there is a good case for DI: When a need arises to decouple a dependency from its caller, or to allow clients to inject custom implementations (e.g., the strategy pattern). In such case, when there is a value, you may consider whether the value of DI is worth its price. If you don’t have this case, why pay for nothing?

I recommend reading the first paragraphs of this blog post ‘Dependency Injection is EVIL’ (and absolutely don’t agree with these bold words)

☀️ Better alternative: ‘Lean-ify’ your engineering approach — avoid using any tool unless it serves a real-world need immediately. Start simple, a dependent class should simply import its dependency and use it — Yeah, using the plain Node.js module system (‘require’). Facing a situation when there is a need to factor dynamic objects? There are a handful of simple patterns, simpler than DI, that you should consider, like ‘if/else’, factory function, and more. Are singletons requested? Consider techniques with lower costs like the module system with factory function. Need to stub/mock for testing? Monkey patching might be better than DI: better clutter your test code a bit than clutter your production code. Have a strong need to hide from an object where its dependencies are coming from? You sure? Use DI!

// humidity-service.ts - not customer facing
export async function getHumidity(when: Datetime): Promise<number> {
  // Fetches from some specific cloud service
}

// weather-service.ts - customer facing
import { getHumidity } from "./humidity-service.ts";
// ✅ No wiring is happening externally, all is flat and explicit. Simple
export async function getWeather(when: Datetime): Promise<number> {
  // Fetch temperature from somewhere and then humidity from GoogleHumidityService
  // Nobody needs to know about it, its an implementation details
  await getHumidity(when);
}

1 min pause: A word or two about me, the author​

My name is Yoni Goldberg, I’m a Node.js developer and consultant. I wrote few code-books like JavaScript testing best practices and Node.js best practices (100,000 stars ✨🥹). That said, my best guide is Node.js testing practices which only few read 😞. I shall release an advanced Node.js testing course soon and also hold workshops for teams. I’m also a core maintainer of Practica.js which is a Node.js starter that creates a production-ready example Node Monorepo solution that is based on the standards and simplicity. It might be your primary option when starting a new Node.js solution

4. Passport.js for token authentication​

💁‍♂️ What is it about: Commonly, you’re in need to issue or/and authenticate JWT tokens. Similarly, you might need to allow login from one single social network like Google/Facebook. When faced with these kinds of needs, Node.js developers rush to the glorious library Passport.js like butterflies are attracted to light

📊 How popular: 1,389,720 weekly downloads

🤔 Why it might be wrong: When tasked with guarding your routes with JWT token — you’re just a few lines of code shy from ticking the goal. Instead of messing up with a new framework, instead of introducing levels of indirections (you call passport, then it calls you), instead of spending time learning new abstractions — use a JWT library directly. Libraries like jsonwebtoken or fast-jwt are simple and well maintained. Have concerns with the security hardening? Good point, your concerns are valid. But would you not get better hardening with a direct understanding of your configuration and flow? Will hiding things behind a framework help? Even if you prefer the hardening of a battle-tested framework, Passport doesn’t handle a handful of security risks like secrets/token, secured user management, DB protection, and more. My point, you probably anyway need fully-featured user and authentication management platforms. Various cloud services and OSS projects, can tick all of those security concerns. Why then start in the first place with a framework that doesn’t satisfy your security needs? It seems like many who opt for Passport.js are not fully aware of which needs are satisfied and which are left open. All of that said, Passport definitely shines when looking for a quick way to support many social login providers

☀️ Better alternative: Is token authentication in order? These few lines of code below might be all you need. You may also glimpse into Practica.js wrapper around these libraries. A real-world project at scale typically need more: supporting async JWT (JWKS), securely manage and rotate the secrets to name a few examples. In this case, OSS solution like [keycloak (https://github.com/keycloak/keycloak) or commercial options like Auth0[https://github.com/auth0] are alternatives to consider

// jwt-middleware.js, a simplified version - Refer to Practica.js to see some more corner cases
const middleware = (req, res, next) => {
    if(!req.headers.authorization){
        res.sendStatus(401)
    }

jwt.verify(req.headers.authorization, options.secret, (err: any, jwtContent: any) => {
      if (err) {
        return res.sendStatus(401);
      }
      req.user = jwtContent.data;
      next();
    });

5. Supertest for integration/API testing​

💁‍♂️ What is it about: When testing against an API (i.e., component, integration, E2E tests), the library supertest provides a sweet syntax that can both detect the web server address, make HTTP call and also assert on the response. Three in one

test("When adding invalid user, then the response is 400", (done) => {
  const request = require("supertest");
  const app = express();
  // Arrange
  const userToAdd = {
    name: undefined,
  };

// Act
  request(app)
    .post("/user")
    .send(userToAdd)
    .expect("Content-Type", /json/)
    .expect(400, done);
  // Assert
  // We already asserted above ☝🏻 as part of the request
});

📊 How popular: 2,717,744 weekly downloads

🤔 Why it might be wrong: You already have your assertion library (Jest? Chai?), it has a great error highlighting and comparison — you trust it. Why code some tests using another assertion syntax? Not to mention, Supertest’s assertion errors are not as descriptive as Jest and Chai. It’s also cumbersome to mix HTTP client + assertion library instead of choosing the best for each mission. Speaking of the best, there are more standard, popular, and better-maintained HTTP clients (like fetch, axios and other friends). Need another reason? Supertest might encourage coupling the tests to Express as it offers a constructor that gets an Express object. This constructor infers the API address automatically (useful when using dynamic test ports). This couples the test to the implementation and won’t work in the case where you wish to run the same tests against a remote process (the API doesn’t live with the tests). My repository ‘Node.js testing best practices’ holds examples of how tests can infer the API port and address

☀️ Better alternative: A popular and standard HTTP client library like Node.js Fetch or Axios. In Practica.js (a Node.js starter that packs many best practices) we use Axios. It allows us to configure a HTTP client that is shared among all the tests: We bake inside a JWT token, headers, and a base URL. Another good pattern that we look at, is making each Microservice generate HTTP client library for its consumers. This brings strong-type experience to the clients, synchronizes the provider-consumer versions and as a bonus — The provider can test itself with the same library that its consumers are using

test("When adding invalid user, then the response is 400 and includes a reason", (done) => {
  const app = express();
  // Arrange
  const userToAdd = {
    name: undefined,
  };

// Act
  const receivedResponse = axios.post(
    `http://localhost:${apiPort}/user`,
    userToAdd
  );
  // Assert
  // ✅ Assertion happens in a dedicated stage and a dedicated library
  expect(receivedResponse).toMatchObject({
    status: 400,
    data: {
      reason: "no-name",
    },
  });
});

6. Fastify decorate for non-request/web utilities​

💁‍♂️ What is it about: Fastify introduces great patterns. Personally, I highly appreciate how it preserves the simplicity of Express while bringing more batteries. One thing that got me wondering is the ‘decorate’ feature which allows placing common utilities/services inside a widely accessible container object. I’m referring here specifically to the case where a cross-cutting concern utility/service is being used. Here is an example:

// An example of a utility that is cross-cutting-concern. Could be logger or anything else
fastify.decorate('metricsService', function (name) {
  fireMetric: () => {
    // My code that sends metrics to the monitoring system
  }
})

fastify.get('/api/orders', async function (request, reply) {
  this.metricsService.fireMetric({name: 'new-request'})
  // Handle the request
})
// my-business-logic.js
exports function calculateSomething(){
  // How to fire a metric?
}

It should be noted that ‘decoration’ is also used to place values (e.g., user) inside a request — this is a slightly different case and a sensible one

📊 How popular: Fastify has 696,122 weekly download and growing rapidly. The decorator concept is part of the framework’s core

🤔 Why it might be wrong: Some services and utilities serve cross-cutting-concern needs and should be accessible from other layers like domain (i.e, business logic, DAL). When placing utilities inside this object, the Fastify object might not be accessible to these layers. You probably don’t want to couple your web framework with your business logic: Consider that some of your business logic and repositories might get invoked from non-REST clients like CRON, MQ, and similar — In these cases, Fastify won’t get involved at all so better not trust it to be your service locator

☀️ Better alternative: A good old Node.js module is a standard way to expose and consume functionality. Need a singleton? Use the module system caching. Need to instantiate a service in correlation with a Fastify life-cycle hook (e.g., DB connection on start)? Call it from that Fastify hook. In the rare case where a highly dynamic and complex instantiation of dependencies is needed — DI is also a (complex) option to consider

// ✅ A simple usage of good old Node.js modules
// metrics-service.js
exports async function fireMetric(name){
  // My code that sends metrics to the monitoring system
}
import {fireMetric} from './metrics-service.js'
fastify.get('/api/orders', async function (request, reply) {
  metricsService.fireMetric({name: 'new-request'})
})
// my-business-logic.js
exports function calculateSomething(){
  metricsService.fireMetric({name: 'new-request'})
}

7. Logging from a catch clause​

💁‍♂️ What is it about: You catch an error somewhere deep in the code (not on the route level), then call logger.error to make this error observable. Seems simple and necessary

try{
    axios.post('https://thatService.io/api/users);
}
catch(error){
    logger.error(error, this, {operation: addNewOrder});
}

📊 How popular: Hard to put my hands on numbers but it’s quite popular, right?

🤔 Why it might be wrong: First, errors should get handled/logged in a central location. Error handling is a critical path. Various catch clauses are likely to behave differently without a centralized and unified behavior. For example, a request might arise to tag all errors with certain metadata, or on top of logging, to also fire a monitoring metric. Applying these requirements in ~100 locations is not a walk in the park. Second, catch clauses should be minimized to particular scenarios. By default, the natural flow of an error is bubbling down to the route/entry-point — from there, it will get forwarded to the error handler. Catch clauses are more verbose and error-prone — therefore it should serve two very specific needs: When one wishes to change the flow based on the error or enrich the error with more information (which is not the case in this example)

☀️ Better alternative: By default, let the error bubble down the layers and get caught by the entry-point global catch (e.g., Express error middleware). In cases when the error should trigger a different flow (e.g., retry) or there is value in enriching the error with more context — use a catch clause. In this case, ensure the .catch code also reports to the error handler

// A case where we wish to retry upon failure
try{
    axios.post('https://thatService.io/api/users);
}
catch(error){
    // ✅ A central location that handles error
    errorHandler.handle(error, this, {operation: addNewOrder});
    callTheUserService(numOfRetries++);
}

8. Use Morgan logger for express web requests​

💁‍♂️ What is it about: In many web apps, you are likely to find a pattern that is being copy-pasted for ages — Using Morgan logger to log requests information:

const express = require("express");
const morgan = require("morgan");
const app = express();
app.use(morgan("combined"));

📊 How popular: 2,901,574 downloads/week

🤔 Why it might be wrong: Wait a second, you already have your main logger, right? Is it Pino? Winston? Something else? Great. Why deal with and configure yet another logger? I do appreciate the HTTP domain-specific language (DSL) of Morgan. The syntax is sweet! But does it justify having two loggers?

☀️ Better alternative: Put your chosen logger in a middleware and log the desired request/response properties:

// ✅ Use your preferred logger for all the tasks
const logger = require("pino")();
app.use((req, res, next) => {
  res.on("finish", () => {
    logger.info(`${req.url} ${res.statusCode}`); // Add other properties here
  });
  next();
});

9. Having conditional code based on NODE_ENV value​

💁‍♂️ What is it about: To differentiate between development vs production configuration, it’s common to set the environment variable NODE_ENV with “production|test”. Doing so allows the various tooling to act differently. For example, some templating engines will cache compiled templates only in production. Beyond tooling, custom applications use this to specify behaviours that are unique to the development or production environment:

if (process.env.NODE_ENV === "production") {
  // This is unlikely to be tested since test runner usually set NODE_ENV=test
  setLogger({ stdout: true, prettyPrint: false });
  // If this code branch above exists, why not add more production-only configurations:
  collectMetrics();
} else {
  setLogger({ splunk: true, prettyPrint: true });
}

📊 How popular: 5,034,323 code results in GitHub when searching for “NODE_ENV”. It doesn’t seem like a rare pattern

🤔 Why it might be wrong: Anytime your code checks whether it’s production or not, this branch won’t get hit by default in some test runner (e.g., Jest set NODE_ENV=test). In any test runner, the developer must remember to test for each possible value of this environment variable. In the example above, collectMetrics() will be tested for the first time in production. Sad smiley. Additionally, putting these conditions opens the door to add more differences between production and the developer machine - when this variable and conditions exists, a developer gets tempted to put some logic for production only. Theoretically, this can be tested: one can set NODE_ENV = "production" in testing and cover the production branches (if she remembers...). But then, if you can test with NODE_ENV='production', what's the point in separating? Just consider everything to be 'production' and avoid this error-prone mental load

☀️ Better alternative: Any code that was written by us, must be tested. This implies avoiding any form of if(production)/else(development) conditions. Wouldn’t anyway developers machine have different surrounding infrastructure than production (e.g., logging system)? They do, the environments are quite difference, but we feel comfortable with it. These infrastructural things are battle-tested, extraneous, and not part of our code. To keep the same code between dev/prod and still use different infrastructure — we put different values in the configuration (not in the code). For example, a typical logger emits JSON in production but in a development machine it emits ‘pretty-print’ colorful lines. To meet this, we set ENV VAR that tells whether what logging style we aim for:

//package.json
"scripts": {
    "start": "LOG_PRETTY_PRINT=false index.js",
    "test": "LOG_PRETTY_PRINT=true jest"
}

//index.js
//✅ No condition, same code for all the environments. The variations are defined externally in config or deployment files
setLogger({prettyPrint: process.env.LOG_PRETTY_PRINT})

Closing​

I hope that these thoughts, at least one of them, made you re-consider adding a new technique to your toolbox. In any case, let’s keep our community vibrant, disruptive and kind. Respectful discussions are almost as important as the event loop. Almost.

Some of my other articles​

• Book: Node.js testing best practices
• Is Prisma better than your ‘traditional’ ORM?
• Book: JavaScript testing best practices
• Practica.js — A Node.js starter
• Node.js best practices

✏️ Written by Michael Solomon and Yoni Goldberg

What are we looking at

The Monorepo market is hot like fire. Weirdly, now when the demand for Monoreps is exploding, one of the leading libraries — Lerna- has just retired. When looking closely, it might not be just a coincidence — With so many disruptive and shiny features brought on by new vendors, Lerna failed to keep up with the pace and stay relevant. This bloom of new tooling gets many confused — What is the right choice for my next project? What should I look at when choosing a Monorepo tool? This post is all about curating this information overload, covering the new tooling, emphasizing what is important, and finally share some recommendations. If you are here for tools and features, you’re in the right place, although you might find yourself on a soul-searching journey to what is your desired development workflow.

This post is concerned with backend-only and Node.js. It also scoped to typical business solutions. If you’re Google/FB developer who is faced with 8,000 packages — sorry, you need special gear. Consequently, monster Monorepo tooling like Bazel is left-out. We will cover here some of the most popular Monorepo tools including Turborepo, Nx, PNPM, Yarn/npm workspace, and Lerna (although it’s not actually maintained anymore — it’s a good baseline for comparison).

Let’s start? When human beings use the term Monorepo, they typically refer to one or more of the following 4 layers below. Each one of them can bring value to your project, each has different consequences, tooling, and features:

Layer 1: Plain old folders to stay on top of your code

With zero tooling and only by having all the Microservice and libraries together in the same root folder, a developer gets great management perks and tons of value: Navigation, search across components, deleting a library instantly, debugging, quickly adding new components. Consider the alternative with multi-repo approach — adding a new component for modularity demands opening and configuring a new GitHub repository. Not just a hassle but also greater chances of developers choosing the short path and including the new code in some semi-relevant existing package. In plain words, zero-tooling Monorepos can increase modularity.

This layer is often overlooked. If your codebase is not huge and the components are highly decoupled (more on this later)— it might be all you need. We’ve seen a handful of successful Monorepo solutions without any special tooling.

With that said, some of the newer tools augment this experience with interesting features:

• Both Turborepo and Nx and also Lerna provide a visual representation of the packages’ dependencies
• Nx allows ‘visibility rules’ which is about enforcing who can use what. Consider, a ‘checkout’ library that should be approached only by the ‘order Microservice’ — deviating from this will result in failure during development (not runtime enforcement)

• Nx workspace generator allows scaffolding out components. Whenever a team member needs to craft a new controller/library/class/Microservice, she just invokes a CLI command which products code based on a community or organization template. This enforces consistency and best practices sharing

Layer 2: Tasks and pipeline to build your code efficiently

Even in a world of autonomous components, there are management tasks that must be applied in a batch like applying a security patch via npm update, running the tests of multiple components that were affected by a change, publish 3 related libraries to name a few examples. All Monorepo tools support this basic functionality of invoking some command over a group of packages. For example, Lerna, Nx, and Turborepo do.

In some projects, invoking a cascading command is all you need. Mostly if each package has an autonomous life cycle and the build process spans a single package (more on this later). In some other types of projects where the workflow demands testing/running and publishing/deploying many packages together — this will end in a terribly slow experience. Consider a solution with hundred of packages that are transpiled and bundled — one might wait minutes for a wide test to run. While it’s not always a great practice to rely on wide/E2E tests, it’s quite common in the wild. This is exactly where the new wave of Monorepo tooling shines — deeply optimizing the build process. I should say this out loud: These tools bring beautiful and innovative build optimizations:

• Parallelization — If two commands or packages are orthogonal to each other, the commands will run in two different threads or processes. Typically your quality control involves testing, lining, license checking, CVE checking — why not parallelize?
• Smart execution plan —Beyond parallelization, the optimized tasks execution order is determined based on many factors. Consider a build that includes A, B, C where A, C depend on B — naively, a build system would wait for B to build and only then run A & C. This can be optimized if we run A & C’s isolated unit tests while building B and not afterward. By running task in parallel as early as possible, the overall execution time is improved — this has a remarkable impact mostly when hosting a high number of components. See below a visualization example of a pipeline improvement

• Detect who is affected by a change — Even on a system with high coupling between packages, it’s usually not necessary to run all packages rather than only those who are affected by a change. What exactly is ‘affected’? Packages/Microservices that depend upon another package that has changed. Some of the toolings can ignore minor changes that are unlikely to break others. This is not a great performance booster but also an amazing testing feature —developers can get quick feedback on whether any of their clients were broken. Both Nx and Turborepo support this feature. Lerna can tell only which of the Monorepo package has changed
• Sub-systems (i.e., projects) — Similarly to ‘affected’ above, modern tooling can realize portions of the graph that are inter-connected (a project or application) while others are not reachable by the component in context (another project) so they know to involve only packages of the relevant group
• Caching — This is a serious speed booster: Nx and Turborepo cache the result/output of tasks and avoid running them again on consequent builds if unnecessary. For example, consider long-running tests of a Microservice, when commanding to re-build this Microservice, the tooling might realize that nothing has changed and the test will get skipped. This is achieved by generating a hashmap of all the dependent resources — if any of these resources haven’t change, then the hashmap will be the same and the task will get skipped. They even cache the stdout of the command, so when you run a cached version it acts like the real thing — consider running 200 tests, seeing all the log statements of the tests, getting results over the terminal in 200 ms, everything acts like ‘real testing while in fact, the tests did not run at all rather the cache!
• Remote caching — Similarly to caching, only by placing the task’s hashmaps and result on a global server so further executions on other team member’s computers will also skip unnecessary tasks. In huge Monorepo projects that rely on E2E tests and must build all packages for development, this can save a great deal of time

Layer 3: Hoist your dependencies to boost npm installation

The speed optimizations that were described above won’t be of help if the bottleneck is the big bull of mud that is called ‘npm install’ (not to criticize, it’s just hard by nature). Take a typical scenario as an example, given dozens of components that should be built, they could easily trigger the installation of thousands of sub-dependencies. Although they use quite similar dependencies (e.g., same logger, same ORM), if the dependency version is not equal then npm will duplicate (the NPM doppelgangers problem) the installation of those packages which might result in a long process.

This is where the workspace line of tools (e.g., Yarn workspace, npm workspaces, PNPM) kicks in and introduces some optimization — Instead of installing dependencies inside each component ‘NODE_MODULES’ folder, it will create one centralized folder and link all the dependencies over there. This can show a tremendous boost in install time for huge projects. On the other hand, if you always focus on one component at a time, installing the packages of a single Microservice/library should not be a concern.

Both Nx and Turborepo can rely on the package manager/workspace to provide this layer of optimizations. In other words, Nx and Turborepo are the layer above the package manager who take care of optimized dependencies installation.

On top of this, Nx introduces one more non-standard, maybe even controversial, technique: There might be only ONE package.json at the root folder of the entire Monorepo. By default, when creating components using Nx, they will not have their own package.json! Instead, all will share the root package.json. Going this way, all the Microservice/libraries share their dependencies and the installation time is improved. Note: It’s possible to create ‘publishable’ components that do have a package.json, it’s just not the default.

I’m concerned here. Sharing dependencies among packages increases the coupling, what if Microservice1 wishes to bump dependency1 version but Microservice2 can’t do this at the moment? Also, package.json is part of Node.js runtime and excluding it from the component root loses important features like package.json main field or ESM exports (telling the clients which files are exposed). I ran some POC with Nx last week and found myself blocked — library B was wadded, I tried to import it from Library A but couldn’t get the ‘import’ statement to specify the right package name. The natural action was to open B’s package.json and check the name, but there is no Package.json… How do I determine its name? Nx docs are great, finally, I found the answer, but I had to spend time learning a new ‘framework’.

Stop for a second: It’s all about your workflow

We deal with tooling and features, but it’s actually meaningless evaluating these options before determining whether your preferred workflow is synchronized or independent (we will discuss this in a few seconds). This upfront fundamental decision will change almost everything.

Consider the following example with 3 components: Library 1 is introducing some major and breaking changes, Microservice1 and Microservice2 depend upon Library1 and should react to those breaking changes. How?

Option A — The synchronized workflow- Going with this development style, all the three components will be developed and deployed in one chunk together. Practically, a developer will code the changes in Library1, test libray1 and also run wide integration/e2e tests that include Microservice1 and Microservice2. When they're ready, the version of all components will get bumped. Finally, they will get deployed together.

Going with this approach, the developer has the chance of seeing the full flow from the client's perspective (Microservice1 and 2), the tests cover not only the library but also through the eyes of the clients who actually use it. On the flip side, it mandates updating all the depend-upon components (could be dozens), doing so increases the risk’s blast radius as more units are affected and should be considered before deployment. Also, working on a large unit of work demands building and testing more things which will slow the build.

Option B — Independent workflow- This style is about working a unit by unit, one bite at a time, and deploy each component independently based on its personal business considerations and priority. This is how it goes: A developer makes the changes in Library1, they must be tested carefully in the scope of Library1. Once she is ready, the SemVer is bumped to a new major and the library is published to a package manager registry (e.g., npm). What about the client Microservices? Well, the team of Microservice2 is super-busy now with other priorities, and skip this update for now (the same thing as we all delay many of our npm updates,). However, Microservice1 is very much interested in this change — The team has to pro-actively update this dependency and grab the latest changes, run the tests and when they are ready, today or next week — deploy it.

Going with the independent workflow, the library author can move much faster because she does not need to take into account 2 or 30 other components — some are coded by different teams. This workflow also forces her to write efficient tests against the library — it’s her only safety net and is likely to end with autonomous components that have low coupling to others. On the other hand, testing in isolation without the client’s perspective loses some dimension of realism. Also, if a single developer has to update 5 units — publishing each individually to the registry and then updating within all the dependencies can be a little tedious.

On the illusion of synchronicity

In distributed systems, it’s not feasible to achieve 100% synchronicity — believing otherwise can lead to design faults. Consider a breaking change in Microservice1, now its client Microservice2 is adapting and ready for the change. These two Microservices are deployed together but due to the nature of Microservices and distributed runtime (e.g., Kubernetes) the deployment of Microservice1 only fail. Now, Microservice2’s code is not aligned with Microservice1 production and we are faced with a production bug. This line of failures can be handled to an extent also with a synchronized workflow — The deployment should orchestrate the rollout of each unit so each one is deployed at a time. Although this approach is doable, it increased the chances of large-scoped rollback and increases deployment fear.

This fundamental decision, synchronized or independent, will determine so many things — Whether performance is an issue or not at all (when working on a single unit), hoisting dependencies or leaving a dedicated node_modules in every package’s folder, and whether to create a local link between packages which is described in the next paragraph.

Layer 4: Link your packages for immediate feedback

When having a Monorepo, there is always the unavoidable dilemma of how to link between the components:

Option 1: Using npm — Each library is a standard npm package and its client installs it via the standards npm commands. Given Microservice1 and Library1, this will end with two copies of Library1: the one inside Microservices1/NODE_MODULES (i.e., the local copy of the consuming Microservice), and the 2nd is the development folder where the team is coding Library1.

Option2: Just a plain folder — With this, Library1 is nothing but a logical module inside a folder that Microservice1,2,3 just locally imports. NPM is not involved here, it’s just code in a dedicated folder. This is for example how Nest.js modules are represented.

With option 1, teams benefit from all the great merits of a package manager — SemVer(!), tooling, standards, etc. However, should one update Library1, the changes won’t get reflected in Microservice1 since it is grabbing its copy from the npm registry and the changes were not published yet. This is a fundamental pain with Monorepo and package managers — one can’t just code over multiple packages and test/run the changes.

With option 2, teams lose all the benefits of a package manager: Every change is propagated immediately to all of the consumers.

How do we bring the good from both worlds (presumably)? Using linking. Lerna, Nx, the various package manager workspaces (Yarn, npm, etc) allow using npm libraries and at the same time link between the clients (e.g., Microservice1) and the library. Under the hood, they created a symbolic link. In development mode, changes are propagated immediately, in deployment time — the copy is grabbed from the registry.

If you’re doing the synchronized workflow, you’re all set. Only now any risky change that is introduced by Library3, must be handled NOW by the 10 Microservices that consume it.

If favoring the independent workflow, this is of course a big concern. Some may call this direct linking style a ‘monolith monorepo’, or maybe a ‘monolitho’. However, when not linking, it’s harder to debug a small issue between the Microservice and the npm library. What I typically do is temporarily link (with npm link) between the packages, debug, code, then finally remove the link.

Nx is taking a slightly more disruptive approach — it is using TypeScript paths to bind between the components. When Microservice1 is importing Library1, to avoid the full local path, it creates a TypeScript mapping between the library name and the full path. But wait a minute, there is no TypeScript in production so how could it work? Well, in serving/bundling time it webpacks and stitches the components together. Not a very standard way of doing Node.js work.

Closing: What should you use?

It’s all about your workflow and architecture — a huge unseen cross-road stands in front of the Monorepo tooling decision.

Scenario A — If your architecture dictates a synchronized workflow where all packages are deployed together, or at least developed in collaboration — then there is a strong need for a rich tool to manage this coupling and boost the performance. In this case, Nx might be a great choice.

For example, if your Microservice must keep the same versioning, or if the team really small and the same people are updating all the components, or if your modularization is not based on package manager but rather on framework-own modules (e.g., Nest.js), if you’re doing frontend where the components inherently are published together, or if your testing strategy relies on E2E mostly — for all of these cases and others, Nx is a tool that was built to enhance the experience of coding many relatively coupled components together. It is a great a sugar coat over systems that are unavoidably big and linked.

If your system is not inherently big or meant to synchronize packages deployment, fancy Monorepo features might increase the coupling between components. The Monorepo pyramid above draws a line between basic features that provide value without coupling components while other layers come with an architectural price to consider. Sometimes climbing up toward the tip is worth the consequences, just make this decision consciously.

Scenario B— If you’re into an independent workflow where each package is developed, tested, and deployed (almost) independently — then inherently there is no need to fancy tools to orchestrate hundreds of packages. Most of the time there is just one package in focus. This calls for picking a leaner and simpler tool — Turborepo. By going this route, Monorepo is not something that affects your architecture, but rather a scoped tool for faster build execution. One specific tool that encourages an independent workflow is Bilt by Gil Tayar, it’s yet to gain enough popularity but it might rise soon and is a great source to learn more about this philosophy of work.

In any scenario, consider workspaces — If you face performance issues that are caused by package installation, then the various workspace tools Yarn/npm/PNPM, can greatly minimize this overhead with a low footprint. That said, if you’re working in an autonomous workflow, smaller are the chances of facing such issues. Don’t just use tools unless there is a pain.

We tried to show the beauty of each and where it shines. If we’re allowed to end this article with an opinionated choice: We greatly believe in an independent and autonomous workflow where the occasional developer of a package can code and deploy fearlessly without messing with dozens of other foreign packages. For this reason, Turborepo will be our favorite tool for the next season. We promise to tell you how it goes.

Bonus: Comparison table

See below a detailed comparison table of the various tools and features:

Preview only, the complete table can be found here

Which Monorepo is right for a Node.js BACKEND now? was originally published in Autodesk TLV on Medium, where people are continuing the conversation by highlighting and responding to this story.

Language review: Jonathan Atia

Why do many teams abandon testing

Production code, the main code where the features live, might not be perfect. That’s a risk or a debt that teams manage in multiple ways. For testing code, the risk is much more dramatic — It can be abandoned altogether. Vanish, gone, not doing anything anymore. Hundreds of coding hours, meetings, and sweat can just get lost.

Have you encountered this? Here is a typical story. 7 pm, late, it’s cold and dark outside. Sarah is a relatively new team member, and she is trying to deploy. All seems excellent except few tests, written by someone else, are failing. Sarah opens the tests file to look through those tests, her eyes can’t believe what they see — each test constitutes 50 lines of code (with Prettier it became 100 loc) including loops, if/else, callbacks. That’s annoying because she thought the challenging part was behind her — Coding the features. Wait, now she also notices that the test is calling other external classes that inherit from more classes and share state with other tests. It’s hard to follow the code traces, her eyes blink tiredly, 8 pm already, Sarah is angry and hungry, did I mention it’s raining outside?. Unlike the tests, the new code seems to work fine — Yes she also tested with Postman and her Browser. Desperately she makes a call to use the magic spell that makes all sorrows end. She adds one little and a sweet word — test.skip. All tests pass now. She deploys!

Guess what happened?

Nothing , it works. The production looks just fine, the tests were probably not updated or wrong.

Wait, what. What is the takeaway here? You might have believed that the worst thing that can happen to the team is a production bug. No, it’s the tests that are being skipped and losing credibility. As a consultant that works with ~20 different teams yearly on various Node.js challenges, I saw this happening frequently. It’s not only me noticing this — There is a known phenomenon called — The eroding testing pyramid, which describes the vanishment of tests from the codebase.

I hear you thinking, this won’t happen to us — We are disciplined, we have a daily scrum meeting, we measure the coverage! None of these will help. Tests just like an orgasm, are very easy to fake (Did I just write this?). These illusional faked testing come in different shapes — Developers might actually write tests but they won’t trust them. Then, in-silent, they will check the code using the browser or POSTMAN to gain real confidence. This manual testing defeats the whole reason for doing testing (speed, automation). Others write tests after the code was written, after the fact, just before production. This workflow implies that there were no tests to serve as a safety net for regression during coding (p.s. It’s not mandatory to write tests before the code, writing them during coding is also fine. Not in the end). With time, the tests become almost useless, a tax that poor developers are asked to pay. Then, with time testing get out of the agenda, they only exist physically like a zombie that lives among us.

What should my team do differently?

Why is this terrible phenomenon happening? Because developers are intelligent, spoily, and efficient creatures that seek the shortest path to the goal. We’re all kept very busy with the main code, the production code (you don’t go to the beach at 2 PM, right?), and have no headroom for additional systems, let alone a testing system. If by chance, there is an additional mental capacity for more code that promises to boost their workflow — it must be tiny and dead-simple. Let me put it loud and clear — Testing can shine only if developers are convinced and happily adopting this habit. This can happen if the tests are not a burden on their busy mind rather a dead-simple technique that brings significant value. The 1st principle of testing is — Plan for outstanding ROI, extracting significant bangs for the buck. Only this will make teams happy to maintain it.

Big words, how can one actually achieve this? By following three principles: First, strive for the minimal amount of testing needed to gain confidence (said also by Kent Beck, ‘father’ of TDD). Second, our tests must resemble the production conditions to provide compelling confidence (see the shift-right and testing-in-production concepts). These two principles will get discussed in a later blog post. The third principle is about writing extremely simple tests — This is the subject of this post.

By saying ‘simple test’, mean truly no-brainer and dead-simple code. Totally flat, complexity level equals one, stand-alone with almost zero dependencies, contains between 7–10 statements, and is written in human-readable language. If you follow this, there is not even a chance that Sarah will decide to skip this test — It’s just too easy. In reality, though, many forces will push developers to deviate from the golden route of simplicity: Tests will include loops and logic because no other way was found to test some behavior. Tests will become long, although the writer really tried to avoid this — There are just too many things to prepare for the test. Other challenges will encourage writing imperative and not-so-simple code. When this happens, stop right there. High tests complexity happens to those who are willing to accept it. The bastards will come to grind you down, but you gotta fight back, and kill it early. To be prepared for this battle, I’ve gathered a coupled of principles and techniques under the acronym BASIC. Let’s go over these principles and then exemplify a ‘nasty’ real-world test packed with unnecessary complexity. Then, let’s apply the BASIC principles over the test and see how it shrinks and turns into a beautiful short test.

The BASIC acronym

Let’s go letter by letter, each represents a principle to consider while testing. Afterward, we will apply these BASIC principles over a long and cumbersome test and turn it into a beautiful short one.

B Black-box — We care only about what the box, our component or object under test, produces and is visible for the caller. Whether it’s API (integration) or code objects (unit). If a bug can not be observed from outside, it won’t matter to our users, do we even care? Avoid testing HOW things work, focus on WHAT they do. It does not matter which functions were called, rather only the outcomes: Publicly noticeable things. For example, the unit under test response, the observed new state, significant calls to external code, and others. By focusing on the outer layer, the amount of details is dramatically reduced and the test writer is pushed to prioritize the important things that might affect the UX. Inherently the test’s length and complexity are lowered.

AAnnotative — A test must have a predictable structure with declarative language. It should feel more like annotations than code. Skimming through a production code is a journey with a clear start and an unknown end — The stack trace might unfold indefinitely and lead the focused reader through many corners of the app. During this time, she might need to stretch her brain to get the intent. Testing should not feel this way rather more like a declarative syntax like … HTML. When reading HTML docs, our mind effortlessly hopes through the various tags, almost expecting what is to come next — Rarely a developer will avoid or fail reading HTML snippets due to complexity.

How can we make the testing experience more like declarative and structured? Use 6 repeatable parts in every test with the AAA pattern (see below), declarative assertions, up to 7 statements. Read JavaScript Testing Best Practices for other similar patterns.

SSingle Door— Every test should focus on one thing. Usually it should trigger one action in the app and check one outcome that happened as a response to this action. My friend Shai Reznik is calling this — ‘Reaction’. An action might be a function call, button click, REST route invocation, new message in a queue, scheduled job start time or any other system event. After performing this action, up to three outcomes might get triggered: Response, change in the state (in-memory or DB), or a call to 3rd party service (Credit: Roy Osherove entry and exit point). In integration tests there might be six types of outcomes, see my checklist here for more details. These potential outcomes are tested using assertions. What is the benefit? Narrowing each test to single action-outcome encourages shorter tests and better root-cause analysis — In case of failure, it’s clear what is not working. This is a good rule of thumb to follow but not too rigidly — It’s OK to test 2 outcomes in a single test, maybe 3, absolutely not 10. Focus on the goal, not the number — Simple and short tests.

I Independent —A test is a 7–10 line long problem, a tiny universe that shares nothing with others. When kept independent, short and declarative — It’s delightful to read by the occasional reader. Beware though, by just coupling a test to some global object — Suddenly it is exposed to side-effects from many other tests and the complexity is dramatically inclined. Coupling can also originate from shared DB records, relying on the execution order, UI state, uncleaned mocking, and others — Avoid all of these. Instead, let each test factor its dependencies, potentially using helpers, including its own DB state to keep it as a self-explanatory universe.

C Copy only what’s necessary— Include all the necessary details that the reader needs to understand within the test. Nothing more, only what’s necessary. Too much duplication will result in unmaintainable tests; on the other hand , extracting important details outside will force the reader to seek pieces of information across many files. As an example, consider a test that should factor 100 lines of input JSON — Pasting this in every test is tedious. Extracting it outside to transferFactory.getJSON() will leave the test vague — Without data, it’s hard to correlate the test result with the cause (“why is it supposed to return 400 status?”). The classic book x-unit patterns named this pattern ‘the mystery guest’ — Something unseen affected our test results, we don’t know what exactly.

We can do better by extracting repeatable long parts outside AND mention explictly which specific details matter to the test. Going with the example above, the test can pass parameters that highlight what is important: transferFactory.getJSON({sender: undefined}). In this example, the reader should immediately infer that the empty sender field is the reason why the test should expect a validation error or any other similar adequate outcome.

Real-world example: A nasty test turns into a beautiful one

Theory aside, let’s transform a ‘nasty’ test with high-complexity into a simple one by applying the five BASIC principles

Code under test

This is the app that will be tested: a service that performs money remittance. The code will validate the money transfer request, apply some logic, ask a banking HTTP service to remit the money and finally persist to DB. This is intentionally not perfectly ‘testable’ code, to say the least—You probably want to learn to test any code, not only ‘perfect’ code.

A nasty test

Let’s test it! This test above checks that when a sender tries to transfer an amount that is bigger than his credit, the declined transfer won’t appear in the user history. In other words, it should not persist. It’s a long and cumbersome test, an anti-pattern, but every line here is allegedly sensible and tackles possible risks. Read it, truly understand the writer’s intent — Every line was included thoughtfully by a great developer who is keen on the details. Only then think about how to improve it. The X symbol means that there is something in that line that could be improved.

Transforming into a beautiful test with the BASIC principles

Let’s go now line by line and remove/fix it, by applying the BASIC principles

Line number 2: Ambuiguiy

❌ Bad (line 2):

test(‘Should fail’, () => {

✅ Better:

describe(‘transferMoney’)// The operation under test

{

  test(‘When the user has not enough credit, then decline the   request’, () => 

}

👩🏼‍🎓 Related pattern: Annotative

🤔 Explanation: What’s wrong here? A test should communicate its intent clearly in a structured and known format as explained in the ‘Annoative’ pattern (follow the 6 repeatable parts pattern)

Line number 3: Mystery

❌ Bad (line 3):

const transferRequest = testHelpers.factorMoneyTransfer({}); 

✅ Better:

const transferRequest = testHelpers.factorMoneyTransfer({credit: 50, transferAmount: 100});//Explictily specified the fields that matter here

👩🏼‍🎓 Related pattern: Copy only what’s necessary

🤔 Explanation: What’s wrong here? The test is about a money transfer without enough credit, this line abstracts the input JSON so there is no clue about how it would look like. The reader has to guess that this JSON would have some issues, but what exactly? Remember, a test is a 7 statements short and punchy story that the reader can understand clearly without leaving the test. A known story-telling technique, Foreshadowing, suggests that all the relevant details should “appears at the beginning of a story, or a chapter, and it helps the reader develop expectations about the upcoming events.”.

Always include within the test the details that matter for the test success or failure. Leave the rest outside. Practically, dynamic factories can greatly help here in factoring default input while getting specific overrides that matter from this test.

✨ Follow me on Twitter

Line number 4: Coupling

❌ Bad (line 4):

serviceUnderTest.options.creditPolicy = ‘zero’; 

✅ Better: Create a dedicated service for every test, configure your own instance, not a global one

👩🏼‍🎓 Related pattern: Independent

🤔 Explanation: What’s wrong here? This line raises the test complexity dramatically. Once all the tests share a global object and can modify its properties, test failures might originate from any other test in the file. Instead of 7 lines long problem you just got a spaghetti of 50 tests to reason about. Maybe some test above set the system state to something that we didn’t plan for? Now instead of investigating a failure within one short test — One must skim through the other test cases in the same file and identify where things went wrong.

Line number 5: Magic numbers

❌ Bad (line 5):

 transferRequest.howMuch = 110; // 

✅ Better:

const amountMoreThanTheUserCredit = 110;
transferRequest.howMuch = amountMoreThanTheUserCredit;

or

transferRequest = {credit:50, howMuch: 100}

👩🏼‍🎓 Related pattern: Copy only what’s necessary

🤔 Explanation: What’s wrong here? Magic number. This line with the arbitarty number ‘110’ tells nothing to the reader, it’s just a number. Is it high? low? what does it represent? The writer meant to pick a value that is higher than the user credit, a critical piece of information that highly relates to the test results, but failed to communicate this. There are multiple techniques that could have told this story better: constant with a descriptive name (e.g. amountMoreThanTheUserCredit) or put the transfer amount nearby the insufficient credit.

Line number 6: White-box testing

❌ Bad (line 6):

const databaseRepositoryMock = sinon.stub(dbRepository, ‘save’);
//This was meant to listen to the ‘save’ function call and ensure it was never called because the transfer should fail

✅ Better:

// Assert — Check that the transfer was not saved using the public API
const senderTransfersHistory = transferServiceUnderTest.getTransfers(transferRequest.sender.name);
expect(senderTransfersHistory).not.toContain(transferRequest);

👩🏼‍🎓 Related pattern: Black-box

🤔 Explanation: What’s wrong here? The writer’s intentions are great, she wants to check that the declined transfer is not accidentally saved to DB, this is what the test is about. The chosen technique is questionable — Usage of mock to listen to the data-access method call and verify that it was not called. Whenever we use test doubles (mocking) out test face two weaknesses: False-positive alarms will fire anytime we legitimately modify the internal call ‘save’ method, although everything still works, the tests will fail. Even worse, our test will lead to positive-false scenarios where the data is saved to the DB using a different path or method, but the test will pass. Instead of checking the natural user flow through the public API, our test is bounded to the internal and increases the maintenance noise. Here is a golden testing rule: Whenever possible, test the user flow as it happens in production — This is what black-box testing is about.

Lines 8–10: Not relevant

❌ Bad (line 6):

expect(transferResponse.currency).toBe(‘dollar’); 
 expect(transferResponse.id).not.toBeNull(); 
 expect(transferResponse.date.getDay()).toBe(new Date().getDay()); 

✅ Better: Just remove as these checks are not part of the tests story

👩🏼‍🎓 Related pattern: Single-door

🤔 Explanation: What’s wrong here? The coder of this test is ambitious and keen to catch any bug in the entire transfer flow. If the transfer currency is not $$ dollar or the response does not contain mandatory fields — Something is wrong. All of this is true and should be tested. But there is one more thing to consider — Tests should be short and focused. If we test a long flow with many outcomes in a single test case, this will result in a less readable test. Once we see a failure, it might be neglectable detail or maybe the entire system is down? Realizing the root-cause will become harder. When testing, it’s just fine to sacrifice some details to emphasize the right trees in a huge forest.

Lines 11: Implementation details

❌ Bad (line 11):

expect(serviceUnderTest.numberOfDeclined).toBe(1);

✅ Better: Remove it, we don’t care about the implementation — If the outcome is satisfactory, the implementation is likely to be fine

👩🏼‍🎓 Related pattern: Black-box

🤔 Explanation: What’s wrong here? It seems like the code internally is making use of a field ‘numberOfDeclined’ to store failures. Probably this will get used to fire metrics or report errors. This assertion checks the implementation of the software, not the publicly available outcome. Inherently, there are much more implementation details than outcomes. Should tests check every function, field, and interaction, it might end with dozens or hundreds of tests per feature, and consequently, much longer test files. Alternatively, when checking only the publicly available result, there are fewer tests and the same issues are caught. Could it be that something internally is wrong but is not reflecting outside? If so, the bug is not affecting the user; Do we even care? Note that checking outcomes does not mean checking only API and UX matters — The ops is also a vital user and checking that metrics, errors, logging are fired is imperative. Those are outcomes, things that the box produces, not implementation details.

Lines 16–17: Overlapping

❌ Bad (line 16–17):

// Let’s get the user transfer history 
const allUserTransfers = serviceUnderTest.getTransfers(transferRequest.sender.name); expect(allUserTransfers).not.toBeNull(); // ❌ Overlapping expect(allUserTransfers).toBeType(‘array’); // ❌ Overlapping

✅ Better:

 expect(senderTransfersHistory).not.toContain(transferRequest);

👩🏼‍🎓 Related pattern: Single-door

🤔 Explanation: What’s wrong here? With great will and motivation, the writer wishes to check that the response array is valid. Great job, it’s just redundant — When asserting that the array does not contain the transfer, all the other checks are validated implicitly. Overlapping assertions is a very popular testing habit, our intuition is to test more, cover more things, if we already wrote a test — Why not include more and more assertions? Because it blurs the story. In testing, strive for less, focus the reader's mind, emphasize the important part. All the more when the same confidence is achieved using less code as demonstrated here.

Lines 20–26: Sighhh imperative code

❌ Bad (line 20–26):

//Assertion: check that declined transfer is not in user history //array ❌
 let transferFound = false;
 allUserTransfers.forEach((transferToCheck) => {
   if (transferToCheck.id === transferRequest.id) {
      transferFound = true;
   }
 });
 expect(transferFound).toBe(false);

✅ Better:

expect(senderTransfersHistory).not.toContain(transferRequest);

👩🏼‍🎓 Related pattern: Annotative

🤔 Explanation: What’s wrong here? The tester wants to ensure that the declined transfer is not saved in the system and is retrievable, so she loops over all the user’s transfers to ensure it does not. Seems legit, isn’t it? No, this is imperative code, one must skim through the implementation to get the intent. If a team goes this route and allows all the programming guns — Loops, if/else, inheritance, try-catch — There is no limit to the potential complexity. A better path is keeping the tests super-simple with declarative (annotative) code — This style involves no implementation details, just read and understand immediately. Practically, stick to declarative assertion libraries (Chai expect, Jest matches). If the community does not offer the desired assertion logic — Write a custom one. Configure linters that prohibit imperative code in testing like jest/no-if, here’s an example video below of issues that test linters can catch. When a complexity checker is in use, allow a complexity level of no more than 1 (totally flat code)

✨ The Result: Short, dead-simple, and beautiful test

Remember that long and cumbersome test that we’ve started with? Here it is now after applying the BASIC principles — Short and concise, isn’t it?

Few words before we say goodbye

The art of testing is also about gaining the right amount of confidence with the minimal effort possible. Even Kent Beck, ‘father of TDD’, the most extremist testing paradigm, wrote once:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence

Minimizing test effort is as important as increasing the test coverage. This post covered how to reduce the time spent on dealing with tests complexity — An important cornet stone in an efficient testing strategy. However, one might still have too much testing or anemic testing — These are the next two achievements to unlock toward testing serenity.

Before you leave, here is a small gift: A poster of the key points that were covered.

Other good reads

• Why Good Developers Write Bad Unit Tests- @michaellynch_29497
• JavaScript Testing Best Practices — Yours truly (shameless plug) Yoni Goldberg
• Write tests. Not too many. Mostly integration — Kent C. Dodds

My testing workshop and courses

✨ Nodeconf — 5 hours advanced Node.js testing workshop next week

✨ My online Node.js testing course

✨ Follow me on Twitter

Short Intro

I’ve compiled below 20 skills, technologies and considerations on choosing between them. Picking the right tools became one of our greatest challenges — the Node.js ecosystem has matured and present attractive options in almost every field. Vanilla or TypeScript? Ava, Mocha or Jest? Express, Fastify or Koa? or maybe Nest? should I include ES6 modules in my next project or stick to good-old ‘require’? Mixing and matching among all of those require deep familiarity with the consequences. These rich set of tools and paradigms also encourages you to tiptoe into unexplored territory. I hope that the next bullets will inspire you to enrich your toolbox and diversify yourself.

My name is Yoni Goldberg, I’m an independent Node.js consultant, the co-author of Node.js best practices, and JavaScript testing best practices. I work with customers in US & Europe on planning, testing and hardening their apps.

Follow me: Twitter, Blog, Newsletter, Testing workshop

📗 Want to take your testing skills to the extreme? Consider visiting my comprehensive course ‘Testing Node.js & JavaScript From A To Z’

Reviewed & improved by Bruno Scheufler

1. Use TypeScript features thoughtfully

TypeScript is exploding: the amazing chart here that shows x5 increase in TypeScript PRs remove any doubt, researches tell that it prevents errors and the community is in love. Though some voices still doubt it (check out this great and well-reasoned post by Eric Elliott) it’s clear that TypeScript has won. Now what? if you look carefully under the hype, TypeScript actually brings two mutual-exclusive offerings to the table: type-safety (including better docs and intellisense) and advanced design constructs. Many teams use TypeScript for better type safety yet unintentionally, without any proper planning, also use its fancy features —generics, abstract classes, interfaces, namespaces, etc. These teams change their design style from vanilla JavaScript to fancy OOP unintentionally due to the ‘law of the instruments’ — a cognitive bias that involves using the tooling in hand whether they are the right choice for the mission or not. In other words, if ‘abstract class’ exists in the toolbox — developers will use it. If you do favor the aforementioned coding techniques— that’s fair, go with this. Just don’t change your code and design style for the wrong reasons

Examples:

• Type Inference can help in reducing TypeScript verbosity and make it look more like vanilla JavaScript
• Type aliases are an alternative simple and leaner way for defining constructs (comparing with interfaces).
• Using TypeScript for encapsulation? access modifiers are coming soon to vanilla JS

2. Modernize your testing toolbox. Ava & Jest are changing the game

Big things are happening in the testing field. According to the recent state of js survey, developers` satisfaction from testing tools increases more than any other domain. A revolution is happening on test runners as well as the good old veteran Mocha and Jasmine are losing the top for the new sophisticated kids in town Jest and Ava. Thanks to the modern approach they bring, it’s possible to test more, cover more ground and find more bugs. Why?

1. Speed — modern test runners are faster thanks to the multi-process execution model. They also apply advanced optimizations like learning tests stats over time and prioritizing slow tests to run first. This is not only about test runners, but other tools also boost test speed: fake in-memory database allow testing with DB without the involvement of IO, some npm packages offer a local and in-memory version of popular cloud services. To name a few examples.
2. Outstanding developer experience — modern test runners and tools are designed to accompany the coding experience and provide valuable insights. For example, should a test fail, Jest and Ava will not just report a failure rather extract the related code from the unit under test. Thanks to this, developers get much richer context which leads to a faster resolution

Some of the traditional tools were designed for CI or for occasional test execution. In times where teams deploy once a day, discovering a bug after 4 hours is not good enough. Modern tooling allows running tests, including component tests with DB, constantly even during coding. This approach allows for testing more layers and more use cases earlier and it’s called ‘shift left’ — read more about it below

Examples:

• AVA & Jest are the new sophisticated kids in town
• mongodb-memory server is amazing for testing with MongoDB server — it installs, instantiates and configures a local and real Mongo with in-memory engine
• Cypress makes E2E testing, including backend’s API related tests, a delightful experience
• aws-sdk-mock fakes many AWS services

3. Plan your ES6 modules usage strategy. See, it’s a bit tricky

The long-awaited ES6 modules were unflagged recently, so you might be tempted to use it right away. They bring great opportunities to Node land like a modern syntax for importing modules, compatibility with frontend syntax (important for package maintainer that need to support both Node and browser runtime) and asynchronous modules resolution that opens the door for top-level async-await and better tree shaking. Cool. However, there are some implications that one must be aware of before jumping on the ESM wagon: not all the supporting features are implemented yet. For example, it’s unclear yet how test doubles libraries like Sinon and Jest can ‘mock’ such modules so your wagon might break on the side of the road with smoke.

Given all of these considerations, what’s your strategy: jump straight into the ESM water and work around the issues? or use ESM with babel/TS as a safety net? maybe keep on with gold old common js ‘require’ but avoid incompatible syntax like usage of __filename, __dirname, JSON resolution, and others? there are no strict answers here but at least we strive to ask the right questions

Thanks to Gil Tayar for these great insights

Examples:

• Differences Between ES Modules and CommonJS from the official Node.js docs
• The ESM roadmap and features that are not implemented yet
• Great post on ESM syntax and status

4. Meet the latest JavaScript features that are turning green soon

I’m not a big fan of chasing for every new language feature, sometimes these shiny toys work against the code simplicity and clarify. From time to time, however, some really valuable Javascript features are presented so it’s worthwhile watching the TC39 proposal list and node.green to identify attractive new features that fit your coding style

Examples:

• Optional chaining is at stage 4 and part of Node.js 13.3 behind a flag. This one is going to catch a lot of popularity. Some love it, others don’t
• Private methods and fields are at stage 3 (active proposals) so if you’re opting for TypeScript only for encapsulation — now there is one more option to choose from
• Nullish coalescing (stage 4) will finally put a stop to our nasty habit of checking for nulls & undefined using the !variableName syntax (which also includes zero and therefore is error-prone)
• Promise.any is the latest edition to the promise.{something} family, unlike all the others (promise.all, promise.settleAll, promise.race) it will abort should any Promise resolve or reject. This is one of the final nails in the coffin of Promises helper libraries like ‘async’ and ‘BlueBird’

5. Experiment architectures outside of your comfort zone. Note how GraphQL is disrupting the traditional models

Great techniques exist in different paradigms that you can embrace without changing your architecture. Also, most companies have a variety of different apps/microservices types like data-driven search and reporting apps, others based on heavy logic and some are just streams of data. Why apply the same treatment to these different requirements? If all you have is a screwdriver, every challenge starts looking like a screw.

Ensure you’re familiar with layered architectures like n-tiers, DDD, Hexagonal/Onion/Clean. They look very different but their primary principle is smilar — isolating the domain (i.e. core data schema and business logic) from the surrounding tech (e.g. APIs, DB). Introduce yourself also to streaming style architectures which see a great increase in popularity. Then, spend some time with data-driven architectures that are best implemented nowadays with GraphQL frameworks.

Speaking of GraphQL, it’s interesting how some of its flavors disrupt the traditional separation between API and data-access layers, instead of repeating similar code and schemas twice these frameworks allow to declaratively define the entire app with one schema. This approach will greatly boost the go-to-market for data-driven apps which are not likely to embed complex logic.

Examples:

• This video by Dev Mastery about Clean Architecture is a gem. It also includes a starter example dev-mastery comments-api
• node-api-boilerplate is a great starter with DDD mindset
• Prisma server allows backend-less data-driven app
• Apollo server is GraphQL-first framework for building apps that is centered around Graph schemas
• express-graphql brings GraphQL merit without forcing to quit the comfy Express environment

6. Check out the winner of the 2019 oscar — Nest.js

After years of Express life, we need a little nest (js). With such amazing growth, you simply can’t ignore it. I would argue that Nest.js is the most remarkable thing that happened to Node.js in 2018/2019— for the first time, we have a full-fledge consensus framework like Java Spring and Python Django. Until 2018, teams without strong design skills had to architect their backends, spend a great time on plumbing and invent the wheel. Being one who engages with ~15 projects every year, believe me, I’ve seen so many types of wheels. Too many. My friend and colleague Gil Tayar funnily adapt the Anna Karenina principle to software: ‘All happy projects are alike; each unhappy project is unhappy in its own way.’

Unlike Express & co., Nest.js brings a full-fledged, batteries-included, framework (e.g. handles the data access layer, validation, etc). Its design style is highly ‘inspired’ by Angular — opinionated, TypeScript-based and embodies heavy modularization constructs. That said, it still offers great flexibility in choosing it’s sub-frameworks. Given all these goodies, I have no single doubt that teams doing their first steps with Node.js will move way faster with Nest.js rather with minimalist Express approach.

With all it’s greatness, it’s not flawless. One may doubt, is the heavy modularized Angular approach that was designed to ease the pain of huge frontend codebase suits the backend needs? aren’t we jumping too far from minimal Express to a huge and such an opinionated framework? are all of these heavy modularity features needed in a world of small Microservices? or the equivalent, isn’t it promoting monoliths (“I can easily handle 30,000 LOC in my code base”)?

At least we now have an option to choose from.

Examples:

• Nest.js official docs
• Quick ‘getting started’ example
• Quick overview of its philosophy and features
• I learned Nest.js using this course which I highly recommend (no affiliation of course)

7. Apply gradual deployment techniques like feature-flagging, canary or traffic shadowing

Have you heard about the epic Cloudflare downtime where a developer who wanted to experiment some feature in production rendered a big part of the internet down? Nothing will boost your confidence and speed than knowing that your deployment engine catches errors before your users do. A bunch of techniques will provide this magic. Each one achieves this in its own way but the overall idea is the same — serve the next version to a limited group of users and measure whether it seems stable. Going with this approach, we actually separating the deployment phase from the release phase. Some say it’s as important as testing, I suggest than anything that measures our pipeline is a TEST.

What are these techniques? Canary is the most well-known and simple. It tunes the routing so the next version is deployed and served to a group of users, starting from users are more likely to tolerate bugs (e.g. office employee, non-paying customers) and as the confidence grows it serves to more and more users. This might sound complex, but frameworks such as istio for K8S and AWS serverless handle most of the heavy lifting work. The next technique, feature flagging, is more powerful but also demands to get your hand dirty. It basically suggests wrapping a feature code with condition criteria that tell which users should benefit this new feature. Usually, it also comes with a dashboard for product managers to turn on and off features. This allows non-technical users to be part of the party and also support finer-grained advanced criteria. For example, using flags you may activate some experimental feature only for users from a specific city, on a specific machine instance that has a specific browser. One last super-interesting technique to look for is traffic shadowing which I’ll leave you to read about.

The value of these techniques is immense: unlike code-testing which demands great effort all the time, tuning your routers for canary deployment only happens once(!) and it also ‘test’ you code under realistic production environment. Learn about this fascinating world and plumb one of this technique into your pipeline

Examples:

• fflip is a minimalist and free feature flagging for Node.js
• Unleash is a rich feature flagging including a dashboard for product folks
• it’s easy to mess your code with feature flagging, read some best practices first
• Kubernetes istio greatly ease canary deployment set up
• With AWS serverless setting a canary is a breeze
• Intro to traffic shadowing

8. Shift your testing left — test more things and sooner

The shift left concept puts forward a sensible claim — the later a bug is discovered, the pricier it is to fix it. Consider a case where you discover a performance issue late on a staging environment, after short analysis it turns out that the fundamental DB data model must be changed — this is likely to incur significant code changes. Some researchers claim it might cost up to x640 times more when a bug is discovered too late on production. In plain words, the traditional model where a developer is focused on unit testing only and then weeks later the QA performs realistic E2E and advanced tests is slow and pricy. This well-known diagram brings this point home safely.

Test more things sooner, discover bugs early. How can we translate this idea into tangible development tasks? run a diversified set of tests as part of every commit and even during coding: component/api tests with real (in-memory?) DB just like you run unit tests, tests with realistic production input using dedicated property-based libraries, apply security scanners, run performance load tests and more. See below a list with dozens of tests one can run across the pipeline

Examples:

• Fast-check npm package for property-based testing allows invoking your units/API with many input permutations
• Check our framework and tools for scanning docker containers for CVEs like snyk, Trivy, Quay
• Tuning your real DB for in-memory operation without IO will make it practical to run api/component tests instantly almost like unit tests: this is how you would tune PostgreSQL (here is a ready-to-use dockerized version), mongodb-memory server will install and configure a local and real Mongo with in-memory engine.

9. Shift your testing right — test in/with production

‘Testing in production’ is a mega-trend in the testing community. It’s based on an idea called ‘shift-right’, which suggests that traditional tests on development and staging environment are less realistic and probably won’t prevent enough issues. The modern production has so many moving parts and parties, so many issues are likely to occur or get discovered only in production. Consequently, many tests must be conducted on the production environment itself for monitoring purposes but also to better test the future versions (e.g. serve some small traffic to the next version). The most straightforward production test is monitoring but many other advanced techniques exist like traffic shadowing, a/b tests (as a technical measure), load testing, tap-compare, soak testing and others.

So should we shift left or right? both. A modern approach for software delivery is not just thinking about tests rather about a pipeline. Given that many phases exist until the next version is served to the user — planning, development, deployment, release — each one is another opportunity to realize issues, stop, or build accumulating confidence. Code testing is a significant step on the pipeline, but plugging other tests into the pipeline will provide more confidence.

Examples:

• Masterpiece post: Testing in production the safe way
• My short video on traffic shadowing test with Node.js
• YouTube — Running E2E tests in production with Node.js

10. Be ready to use your new async pocket knife — worker threads

The most popular Node.js interview question might vanish soon: ‘is Node.js really single-threaded?’. As of version 11.7, we welcome a new family member in the async toolbox — worker thread. This tool, unlike any other, can address a very painful blindspot in Node. If 100% of the requests are CPU-intensive — no web framework, including Go & Java ones, can help tame this beast. However, a more popular workload is when only 1–10% of requests grabs the CPU for long time — most of the non-Node frameworks will prevent this automatically (thread per request). Node.js couldn’t — when serving 1000 req/sec, it’s enough for 1 to be CPU-intensive so all the other 999 suffer. There was no remedy to this pain, child process for examples are too slow to spin-up and can’t share memory. Good news, this is now tamable: worker threads can spin-up a dedicated event loop so the main one will remain snappy.

Now for some bad news — worker threads are not a lightweight thread that one spawn in no-time on demand. They actually duplicate the entire engine so it can become quite slow until they start running CPU-bounded requests will suffer additional delay. For this reason, consider a thread-pool (link below)

Examples:

• Quick hello-world
• An exhaustive intro to worker threads
• The official doc is quite short and beneficial for skimming through the API
• Thread pool implementation example

11. Deepen your Docker and Kubernetes understanding. It highly affects your Node.js code.

The DevOps storm means different things to different teams. For some, it is about making Dev perform also Ops work (e.g. being on-call), for others it’s more of a recommendation to plan early for production. At a minimum, it’s expected from developers to understand the production run-time as it highly affects the coding decisions and patterns. Mostly the decisions that sit on the intersection between Dev and Ops.

Few examples: it’s a well-known practice to ensure all outgoing requests are being replayed upon failures (the circuit breaker pattern), this can be done both on the infrastructure level using K8S Istio, or at the code itself using dedicated packages. Which one would you prefer and why? interesting choice, isn’t it? Let’s discuss other scenarios — K8S might kill and relocate pods, when it sends a kill signal the webserver might handle 2000 users. When a pod just crashes, they will quickly become angry 2000 users unless you implement a graceful and thoughtful shutdown. What is the grace period? well, this requires some Kubernetes learning, right? Sometimes the kill signals from K8S won’t even reach to your code if you use ‘npm start’ command, why? this requires some understanding of how Docker processes and signals are managed (the 1st link below will answer this question). One other interesting challenge is settling two contradicting things — how can test tools run within Docker containers during the pipeline but then removed before production? One last interesting example is configuring the allowed memory per container, given that v8 recently stopped limiting the heap size which will now keep growing as needed, this might interfere with K8S resource limits (common best practice) — make a decision and align the Node.js side with the K8S side

all of these challenges call you to dig deeper into the fascinating world of Docker clusters (or Serverless if you wish)

Examples:

• A masterpiece video by Bret Fisher on Node.js Docker best practices. Don’t miss this one!
• istio circuit breaker vs npm code-level circuit breaker
• Nice article on managing Node.js memory in a containerized environment by Ravali Yatham

12. Security: Learn to think like an attacker by skimming through vulnerable code examples

If you can’t think like an attacker, you can’t think like a defender. In 2020, you shouldn’t outsource the defense work to third-party companies or rely solely on static security scanners: The amount of attack types is overwhelming (The development pipeline and NPM are the latest trends). Developers training is the key — bake security DNA into yourself and your team and add a security touch to everything. A useful way to deepen your security understanding is to go through examples of vulnerable code and attack vectors. See below few example links that might greatly help

Examples:

• NodeGoat is a project that intentionally embodies security weaknesses for educational purposes. Don’t miss this doc page with attack examples
• Read my list of Node.js Security best practices which contain 23+ attack ideas including JavaScript code examples
• Conduct a monthly threats analysis meeting where the team tries to look at the application design and propose attacks. sound boring? not necessarily if you add some gamification and reward members that find an exploit, or run a competition between a blue team that designs a module vs the read team which tries to find exploits

13. Learn at least one: ELK or Prometheus

Monitoring is a crucial set up that should be well hardened and demands cooperation between Ops and Dev. No monitoring solution can be perfect without developers' involvement. These two popular monitoring systems, ELK and Prometheus, sound like sys admin toys but in fact, developers can learn a lot by configuring them. In any case, the mandatory activity for developers is being involved in exposing the metrics.

Ops folks know nothing about the event loop and how to monitor it (npm package does this), only you can propose and implement this important metric. Only developers can suggest the right V8 monitoring limits alerts. Developers might even write automated tests to ensure that when application errors are thrown — the right metrics are incremented. One another valuable activity is custom applicative metrics — coding some measurement of user activities can be very efficient in tracking production anomalies. Consider an e-commerce app, if the number of purchases is tracked and it suddenly drops dramatically in production — this is likely to imply some underlying issue

Examples:

• A post about first steps with Prometheus and Node.js
• Prometheus client library for Node.js — a great resource to learn about the metrics that are exposed by default
• A post about first steps with Node.js & ELK
• A comprehensive overview of ELK
• Read about the 4 monitoring ‘Golden signals’
• Read Google Site Reliability Engineering book, or at least the chapter on monitoring

14. Use machine learning as a black-box product

This bullet is targeted for machine learning (ML) newbies who work for products that don’t rely heavily on ML. If you aren’t one of those — feel free to step forward to the next bullet. So you’re like me clueless about ML, that’s fair, we intentionally chose not to build expertise on this field. Still, we can do much better if we just understand common ML needs and solutions so we consume those once a need arise. The JavaScript ML world has matured to a point where there are many stable libraries that can produce great value without intimate knowledge of the implementation. If we just understand WHAT they’re doing (+ how to configure )— we gain a special hammer to our toolbox. When this might prove to be valuable? say you sit in a meeting and the product manager mention something about organizing data (e.g. customers) into groups — suggest using cluster algorithms. Feeding the data into an adequate library might be enough to extract great insights. Often you’ll need to gain more knowledge to configure it correctly — that’s fair, hire an expert, become an expert, at least you knew where to start and structured the solution path. Does someone now ask about recommending things to the user? classify things? Find similarities between texts? predict? analyze audio or image? there is a lib for each one of those — pull out your weapon at the right moment

Obviously, better delve into the details and read the manuals. In no way, I’m suggesting a careless use of technologies. What I do propose however is that becoming familiar with the high-levels is better than knowing nothing and can build the motivation to keep exploring. Should I used ML taxonomy wrong above — please understand, I’ve just started my ML journey

Examples:

• ml.js — a swiss army knife of machine learning tools
• Brain.js specializes in neural networks, has great docs and a free video course
• Natural brings the world of NLP into Node land: text comparison and similarities, sentiment analysis, text classification and more

15. Sleep >7 hours a night. This matters far more than any technology you use and scientifically prooved to make you a better developer

This comes from Hillel Wayne and it’s one of the best tweets I’ve encountered in the last year :

your sleep quality and stress level matter far, far more than the languages you use or the practices you follow. Nothing else comes close: not type systems, not TDD, not formal methods, not ANYTHING.

It’s packed with examples and researches — I urge you to visit there and dig into those pearls of wisdom

Examples:

• Many software paradigms are not proven to be useful — TDD research here, clean architecture link. Other examples can be found within the Tweet
• Research: No clear evidence that a chosen programming language makes a different
• Research: Chronically getting less than 7 hours of sleep causes dramatic degradation on all mental tests

16. Quit Express, it’s aged and not maintained properly. Fastify and Koa are great candidates in 2020

Did you remember to wrap all your Express routes with try-catch, then move to the next() method and finally return some appropriate status to the user? If no, your process will crash without a trace. If yes, you just spent a great time on plumbing straight-forward pieces that add no value to your business. Isn’t this what frameworks are here for? Though Fastify & Koa won’t handle all the error paths for you (e.g. uncaught exceptions) they address this with a modern approach that requires less effort. They both also natively support async routes. Those are just a few examples where a modern and maintained framework could do better for you.

The last commit to the Express project was pushed somewhere 6 months ago… Since then Fastify & Koa saw dozens of builds and they keep improving. It’s frankly not appropriate for the Node.js ecosystem to rely heavily on a library that doesn’t keep in step with the times.

That said, most of the community tools and docs rely on Express. I hope to see community leaders, course makers, bloggers creating more content on its modern alternatives.

Examples:

• Fastify
• Koa

17. Revisit these bullets from last year — some are still highly relevant

I’ve published a similar post in 2019 and many of the bullets seem important also in 2020. Here are some specific recommendations:

Examples:

• Bullet number #4 — “Plan how to utilize Async-Hooks to reach better tracing and context”
• Bullet number #11 — “Have a package update strategy. A lesson learned in 2018: updating too soon is a dangerous practice”
• Bullet number #17 — “Deepen your Linux OS understanding, focus on the anatomy of a Linux process”
• Bullet number #18 — “Dive deeper into the Node.js internals”

18. Enrich your CI with automated quality tools

Our journey for quality and safe deploys is usually centered around testing. The caveat of code-based testing (E.g. unit tests) is their price — every new functionality demands writing more testing code. This usually pays off but still painful. Some tools like linters, scanners, and static analyzers offer a different deal — for a one-time setup they will discover bugs forever. This is a great opportunity for lowering the price of building confidence, almost a free lunch. The list of tools is growing every year so keep following and enrich your CI — below I’ve included a few examples of modern tools.

Examples:

• Scan Docker containers for CVEs using tools like snyk, Trivy, Quay
• lockfile-lint will detect attempts to inject malicious dependencies using the npm lock file (e.g. edit a dependency URL within the lockfile)
• swagger-express-validator will ensure that your routes conform to the swagger schema
• eslint-plugin-import is an outstanding linter that performs dozens of checks on module and dependencies resolution. For example, it can warn when import/require is not at the beginning, disallow mutable exports with var or let, discoverextraneous packages that aren't declared with package.json and much more
• commitlint will enforce semantic commits which then can lead to automatic semantic versioning of Microservice and packages
• dependency-cruiser is a CLI tool that allows declaring flexible policies on allowed and disallowed dependencies. Using this you can craft custom rules like what licenses are allowed, allowed paths and more

19. Enrich your mindset, diversify your toolbox

We often surround ourselves with favorite technologies and ignore alternatives based on prejudice. Here are some typical sentences I hear in my network: ‘Functional programming is not practical’, ‘REST API is dead, ‘TDD is not for me’, ‘ORMs are evil’, ‘TypeScript is too verbose’.

These are false-dichotomies — it’s not a binary question, all these paradigms embody many different ideas, and still, we tend to pick all or ignore all. For example, Functional Programming currying and monads feel weird? that’s fair, consider other more mainstream FP ideas like pure functions. Ignoring TypeScript because OOP is not your style? maybe use only its type system and stick to vanilla JS objects and functions. Cherry-pick ideas and features, not a package.

How exactly do I plan to diversify my stack in 2020? I’ll probably use Fastify for the web layer, with a mix of REST and GraphQL. The data access layer will constitute some lightweight ‘ORM’ but only for migrations and connection pooling (no dichotomy, I’m picking the right ORM features for me). As for the DB, I plan to use a relational DB mixed with JSON columns. My coding style is based on simple and flat vanilla JavaScript objects — but I usually mix it with some classes when appropriate and keep most of my functions pure. The overall architecture style will be centered around Microservices but maybe in a monorepo — I’m not obliged to pick all the Microservice bells and whistles, right?. As of testing, I definitely want to run TDD-style iterations of refactoring for my code, but not necessarily code the tests first — I pick the TDD features that suit my style. What type of testing? mostly component tests (i.e. api) but mix this with unit testing to cover parts with heavy logic.

By no mean, I suggest that this is the best stack. It is, however, a diversified stack that mixes and matches multiple ideas from many paradigms. Obviously assemble your own stack, just don’t be afraid of tiptoeing into unexplored territory and get inspiration from many sources of wisdom.

p.s. I’m not advocating for becoming a jack of too many trades. Actually mastering some technologies is important. My point is being pragmatic and open to great ideas. Don’t be “that screwdriver guy”, enrich your mindset, diversify your toolbox

20. Get inspiration from these great 5 starter projects

Starter (boilerplate) projects are a genuine source for knowledge — just skim through the code for 10–20 min and get many ideas to embrace. I’ve packed here below some quality starters, each brings some unique approach so you enrich your mindset with new paradigms

Examples:

• node-api-boilerplate is a great showcase for DDD with an application layer that demonstrates organizing code by feature
• dev-mastery comments-api is an excellent translation of clean architecture to Node.js and it comes with this super explainer video
• Typescript-starter won’t teach you any architecture concepts but a nice showcase for TypeScript set up with modern libraries and great docs
• nodejs-api-starter is the starter for those who seek a real-world GraphQL implementation including errors and auth
• bulletproof-nodejs is a well-known starter that uses the 3-tier folder structure which is my personal and recommended way of structuring a Node.js app

📗 Liked the content here and want to get 10+ hours course on Node.js quality and testing? Visit my online course ‘Testing Node.js & JavaScript From A To Z’

Thank You. Other articles you might like

• 23 Node.js security best practices
• 45 JavaScript testing best Practice
• 85 Node.js best practices
• My testing workshop

⭐ Want more? follow me on Twitter⭐

Often when considering, or working with Microservices, we are uncertain whether some component should be a Microservice. This component might be a mailer, price calculator, facebook connector, message queue facade, or anything that seems to benefit from being an autonomous component.

The checklist below means to give a sense of whether you need a Microservice or maybe it’s probably an overkill tool for your needs and you can deploy it as a package or maybe even just another class/object.

Although all the reasons below are valid and essential to consider, take the ‘gamification’ and scoring part with a sense of humor.

Answer all the questions and sum your score

1. Start here — Start with zero, add or subtract points based on the questions below

Start with 0️⃣

Move to the next question 👇

2. Different teams: Will a different dedicated team develop this component?

If YES: ➕ add 10 points

Move to the next question 👇

3. Dedicated technology: Is the component likely to be developed using a different programming language?

If YES: ➕ add 20 points

Move to the next question 👇

4. Simplicity: When not extracted out, do things get too complex to “fit in your head”?

If YES: ➕ add 10points

Move to the next question 👇

5. Failure isolation: Should and can this component stay alive and provide value when its collaborators crashed?

If YES: ➕ add 7 points

Move to the next question 👇

6. Operational complexity: Do you already have a Microservice infrastructure?

If YES: ➕ add 5 points

Move to the next question 👇

8. Granular deployment: Is this functionality likely to be updated at a different pace than other components so you can update it without the need to update its collaborators?

If YES: ➕ add 12 points

Move to the next question 👇

9. Stateful service: Does it expose API, DB or any other stateful activity?

If YES: ➕ add 3 points

Move to the next question 👇

10. Ubiquitous language: Is this functionality represents an isolated product domain like a stand-alone feature or having a dedicated product manager?

If YES: ➕ add 3 points

Move to the next question 👇

11. Independent scale: Is this component’s traction is likely to be different than others and might need to scale independently?

If YES: ➕ add 20 points

Move to the next question 👇

12. Dedicated infrastructure: Does the component deserver a unique hardware spec like GPU or extensive RAM support?

If YES: ➕ add 20 points

Move to the next question 👇

13. Change protection: Might this component get replaced sometime soon by another implementation, for example by a 3rd party product?

If YES: ➕ add 3 points

Move to the next question 👇

14. Avoid chatty flow: Is this functionality invoked multiple times during each flow (e.g. log, trace, etc)?

If YES: ➖ subtract(!) 20 points

Move to calculate your results 👇

🗒️ Results — should it be a Microservice?

0–5 points

😂 Seriously? a function might be enough to satisfy your needs

5–10

🤦‍♂️ Oh No. Don’t be a drama queen, read about package managers, modules and other leaner isolation techniques

10–15

🤔 Hmm. If you don’t already have a Microservice infrastructure the ROI is likely to be negative

15–20

👍 Maybe. The value doesn’t seem critical — at least keep the overhead low

20–30

👌 Probably yes, mostly if it’s not your first Microservice

30–50

✌ Oh yes! This component by itself is a good reason to move toward a Microservice architecture

Final wording

Obviously, the scoring is just a nice game which shouldn’t be taken too seriously 😀

I’m curious, did the score made sense? feel free to suggest improvements

Stay in touch

Follow me on Twitter ⭐️

Node.js best practices — 79 best practices for a robust Node application

45+ JavaScript testing best practices

⚠️🎊📢 Important Note: This Article became a GitHub repository with additional 15 best practices and community discussions

If you’re looking for the most recent and comprehensive version — I suggest switching to the GitHub version

Short Intro

As an independent Node.js consultant I’m engaged and review 10+ projects every year and my customers justifiably ask to focus on testing. A few months ago, I started to document here the insights and repeating errors I observe at the field and suddenly it piled-up into 30 testing best practices.

The ideas below span topics like choosing the right test types, coding them correctly, measuring their effectiveness and hosting them in a CI/CD pipeline in the right way. Some examples are illustrated using Jest, others with Mocha — this post is less about tooling and more about the right approach and techniques.

My name is Yoni Goldberg, an independent Node.JS consultant and the co-author of Node.js best practices. I work with customers at the USA, Europe, and Israel on polishing their Node.js applications. Among my service are also test planning, test review, and CI/CD setup. Follow me on Twitter ⭐️

📗 Want to take your testing skills to the extreme? Consider visiting my comprehensive course ‘Testing Node.js & JavaScript From A To Z’

Reviewed & improved by Bruno Scheufler 🙏

⚪️ 0. The Golden Rule: Testing must stay dead -simple and clear as day

Are you familiar with that smiley friend, family member or maybe a movie character who is always available to spare his good working hands, 24/7 assisting when you need him tuned with positive energy yet asking so little for himself? This how a testing code should be designed — easy, valuable and fun. This can be achieved by selectively cherry-picking techniques, tools and test targets that are cost-effective and provide great ROI. Test only as much as needed, strive to keep it nimble, sometimes it’s even worth dropping some tests and trade reliability for agility and simplicity.

Testing should not be treated as a traditional application code — a typical team is challenged with maintaining its main application anyway (the features we code and sell), it could not tolerate additional complex “project”. Should testing grow to be an additional source of pain — it will get abandoned or greatly slow down the development.

In that sense, the testing code must stay dead-simple, with minimal dependencies, abstractions, and levels of indirection. One should look at a test and get the intent instantly. Most of the advice below are derivatives of this principle

***Section 1️⃣ : The Test Anatomy***

⚪ ️ 1. Include 3 parts in each test name

✅ Do: A test report should tell whether the current application revision satisfies the requirements for the people who are not necessarily familiar with the code: the tester, the DevOps engineer who is deploying and the future you two years from now. This can be achieved best if the tests speak at the requirements level and include 3 parts:

(1) What is being tested? For example, the ProductsService.addNewProduct method

(2) Under what circumstances and scenario? For example, no price is passed to the method

(3) What is the expected result? For example, the new product is not approved

❎ Otherwise: A deployment just failed, a test named “Add product” failed. Does this tell you what exactly is malfunctioning?

☺️ Doing It Right Example: A test name that constitutes 3 parts

️☺ Doing It Right Example: The test report resembles the requirements document

⚪️ 2. Describe expectations in a product language: use BDD-style assertions

✅ Do: Coding your tests in a declarative-style allows the reader to get the grab instantly without spending even a single brain-CPU cycle. When you write an imperative code that is packed with conditional logic the reader is thrown away to an effortful mental mood. In that sense, code the expectation in a human-like language, declarative BDD style using expect or should and not using custom code. If Chai & Jest don’t include the desired assertion and it’s highly repeatable, consider extending Jest matcher (Jest) or writing a custom Chai plugin

❎ Otherwise: The team will write less test and decorate the annoying ones with .skip()

😕 Anti Pattern Example: The reader must skim through not so short, and imperative code just to get the test story

☺️ Doing It Right Example: Skimming through the following declarative test is a breeze

⚪️ 3. Lint with testing-dedicated plugins

✅ Do: A set of ESLint plugins were built specifically for inspecting the tests code patterns and discover issues. For example, eslint-plugin-mocha will warn when a test is written at the global level (not a son of a describe() statement) or when tests are skipped which might lead to a false belief that all tests are passing. Similarly, eslint-plugin-jest can, for example, warn when a test has no assertions at all (not checking anything)

❎ Otherwise: Seeing 90% code coverage and 100% green tests will make your face wear a big smile only until you realize that many tests aren’t asserting for anything and many test suites were just skipped. Hopefully, you didn’t deploy anything based on this false observation

😕 Anti Pattern Example: A test case full of errors, luckily all are caught by Linters

⚪️ 4. Stick to black-box testing: Test only public methods

✅ Do: Testing the internals brings huge overhead for almost nothing. If your code/API deliver the right results, should you really invest your next 3 hours in testing HOW it worked internally and then maintain these fragile tests? Whenever a public behavior is checked, the private implementation is also implicitly tested and your tests will break only if there is a certain problem (e.g. wrong output). This approach is also referred to as behavioral testing. On the other side, should you test the internals (white box approach) — your focus shifts from planning the component outcome to nitty-gritty details and your test might break because of minor code refactors although the results are fine- this dramatically increases the maintenance burden

❎ Otherwise: Your test behaves like the child who cries wolf: shoot out loud false-positive cries (e.g., A test fails because a private variable name was changed). Unsurprisingly, people will soon start to ignore the CI notifications until someday a real bug will get ignored…

😕 Anti Pattern Example: A test case is testing the internals for no good reason

️️5. Choose the right test doubles: Avoid mocks in favor of stubs and spies

✅ Do: Test doubles are a necessary evil because they are coupled to the application internals, yet some provide an immense value (Read here a reminder about test doubles: mocks vs stubs vs spies). However, the various techniques were not born equal: some of them, spies and stubs, are focused on testing the requirements but as an inevitable side-effect they also slightly touch the internals. Mocks, on the contrary side, are focused on testing the internals — this brings huge overhead as explained in the bullet “Stick to black box testing”.

Before using test doubles, ask a very simple question: Do I use it to test functionality that appears, or could appear, in the requirements document? If no, it’s a smell of white-box testing.

For example, if you want to test what your app behaves reasonably when the payment service is down, you might stub the payment service and trigger some ‘No Response’ return to ensure that the unit under test returns the right value. This checks our application behavior/response/outcome under certain scenarios. You might also use a spy to assert that an email was sent when that service is down — this is again a behavioral check which is likely to appear in a requirements doc (“Send an email if payment couldn’t be saved”). On the flip side, if you mock the Payment service and ensure that it was called with the right JavaScript types — then your test is focused on internal things that got nothing with the application functionality and are likely to change frequently

❎ Otherwise: Any refactoring of code mandates searching for all the mocks in the code and updating accordingly. Tests become a burden rather than a helpful friend

😕 Anti-pattern example: Mocks focus on the internals

☺️ Doing It Right Example: spies are focused on testing the requirements but as a side-effect are unavoidably touching to the internals

⚪️ 6. Don’t “foo”, use realistic input data

✅ Do: Often production bugs are revealed under some very specific and surprising input — the more realistic the test input is, the greater the chances are to catch bugs early. Use dedicated libraries like Faker to generate pseudo-real data that resembles the variety and form of production data. For example, such libraries will generate random yet realistic phone numbers, usernames, credit card, company names, and even ‘lorem ipsum’ text. Consider even importing real data from your production environment and use in your tests. Want to take it to the next level? see next bullet (property-based testing)

❎ Otherwise: All your development testing will falsely seem green when you use synthetic inputs like “Foo” but then production might turn red when a hacker passes-in a nasty string like “@3e2ddsf . ##’ 1 fdsfds . fds432 AAAA”

😕 Anti-Pattern Example: A test suite that passes due to non-realistic data

☺Doing It Right Example: Randomizing realistic input

📗 Liked the content here and want more?

Get 10+ hours course on Node.js quality and testing? Visit my online course ‘Testing Node.js & JavaScript From A To Z’

7. Test many input combinations using Property-based testing

✅ Do: Typically we choose a few input samples for each test. Even when the input format resembles real-world data (see bullet ‘Don’t foo’), we cover only a few input combinations (method(‘’, true, 1), method(“string” , false” , 0)), However, in production, an API that is called with 5 parameters can be invoked with thousands of different permutations, one of them might render our process down (see Fuzz Testing). What if you could write a single test that sends 1000 permutations of different inputs automatically and catches for which input our code fails to return the right response? Property-based testing is a technique that does exactly that: by sending all the possible input combinations to your unit under test it increases the serendipity of finding a bug. For example, given a method — addNewProduct(id, name, isDiscount) — the supporting libraries will call this method with many combinations of (number, string, boolean) like (1, “iPhone”, false), (2, “Galaxy”, true). You can run property-based testing using your favorite test runner (Mocha, Jest, etc) using libraries like js-verify or testcheck (much better documentation). Update: Nicolas Dubien suggests in the comments below to checkout fast-check which seems to offer some additional features and also to be actively maintained

❎ Otherwise: Unconsciously, you choose the test inputs that cover only code paths that work well. Unfortunately, this decreases the efficiency of testing as a vehicle to expose bugs

☺ Doing It Right Example: Testing many input permutations with “mocha-testcheck”

⚪️ 8. Stay within the test: Minimize external helpers and abstractions

✅ Do: By now, it’s probably obvious that I’m advocating for dead-simple tests: The team can’t afford another software project that demands a mental effort to understand the code. Michael Lync explains this in his great post:

Good production code is well-factored; good test code is obvious…When you write a test, think about the next developer who will see the test break. They don’t want to read your entire test suite, and they certainly don’t want to read a whole inheritance tree of test utilities.

Let the reader get the whole story without leaving the test, minimize utils, hooks, or any external effect on a test case. Too many repetitions and copy-pasting? OK, a test can leave with one external helper and stay obvious. But when it grows to three and four helpers and hooks it implies that it a complex structure is slowly forming

❎ Otherwise: Suddenly found yourself with 4 helpers per test suite, 2 of them inheriting from base util, a lot of setup and tearing-up hooks? congratulation, you just won another challenging project to maintain, you might write tests soon against your test suite

😕 Anti-Pattern Example: Fancy and indirect test structure. Do you understand the test case without navigating to external dependencies?

☺ Doing It Right Example: A test you may understand without hopping through different files

⚪️ 9. Avoid global test fixtures and seeds, add data per-test

✅ Do: Going by the golden rule (bullet 0), each test should add and act on its own set of DB rows to prevent coupling and easily reason about the test flow. In reality, this is often violated by testers who seed the DB with data before running the tests (also known as ‘test fixture’) for the sake of performance improvement. While performance is indeed a valid concern — it can be mitigated (see “Component testing” bullet), however, test complexity is a much painful sorrow that should govern other considerations most of the time. Practically, make each test case explicitly add the DB records it needs and act only on those records. If performance becomes a critical concern — a balanced compromise might come in the form of seeding the only suite of tests that are not mutating data (e.g. queries)

❎ Otherwise: Few tests fail, a deployment is aborted, our team is going to spend precious time now, do we have a bug? let’s investigate, oh no — it seems that two tests were mutating the same seed data

😕 Anti Pattern Example: tests are not independent and rely on some global hook to feed global DB data

☺Doing It Right Example: We can stay within the test, each test acts on its own set of data

⚪️ 10. Don’t catch errors, expect them

✅ Do: When trying to assert that some input triggers an error, it might look right to use try-catch-finally and assert that the catch clause was entered. The result is an awkward and verbose test case (example below) that hides the simple test intent and the result expectations

A more elegant alternative is the using the one-line dedicated Chai assertion: expect(method).to.throw (or in Jest: expect(method).toThrow()). It’s absolutely mandatory to also ensure the exception contains a property that tells the error type, otherwise given just a generic error the application won’t be able to do much rather than show a disappointing message to the user

❎ Otherwise: It will be challenging to infer from the test reports (e.g. CI reports) what went wrong

😕 Anti-pattern Example: A long test case that tries to assert the existence of error with try-catch

☺Doing It Right Example: A human-readable expectation that could be understood easily, maybe even by QA or technical PM

⚪️10. Tag your tests

✅ Do: Different tests must run on different scenarios: quick smoke, IO-less, tests should run when a developer saves or commits a file, full end-to-end tests usually run when a new pull request is submitted, etc. This can be achieved by tagging tests with keywords like #cold #api #sanity so you can grep with your testing harness and invoke the desired subset. For example, this is how you would invoke only the sanity test group with Mocha: mocha — grep ‘sanity’

❎ Otherwise: Running all the tests, including tests that perform dozens of DB queries, any time a developer makes a small change can be extremely slow and keeps developers away from running tests

☺️ Doing It Right Example: Tagging tests as ‘#cold-test’ allows the test runner to execute only fast tests (Cold===quick tests that are doing no IO and can be executed frequently even as the developer is typing)

11. Other generic good testing hygiene

✅ Do: This post is focused on testing advice that is related to, or at least can be exemplified with Node JS. This bullet, however, groups few non-Node related tips that are well-known

Learn and practice TDD principles — they are extremely valuable for many but don’t get intimidated if they don’t fit your style, you’re not the only one. Consider writing the tests before the code in a red-green-refactor style, ensure each test checks exactly one thing, when you find a bug — before fixing write a test that will detect this bug in the future, let each test fail at least once before turning green, avoid any dependency on the environment (paths, OS, etc)

❎ Otherwise: You‘ll miss pearls of wisdom that were collected for decades

***Section 2️⃣ : Test Types***

⚪️ 12. Enrich your testing portfolio: Look beyond unit tests and the pyramid

✅ Do: The testing pyramid, though 10 years old, is a great and relevant model that suggests three testing types and influences most developers’ testing strategy. At the same time, more than a handful of shiny new testing techniques emerged and are hiding in the shadows of the testing pyramid. Given all the dramatic changes that we’ve seen in the recent 10 years (Microservices, cloud, serverless), is it even possible that one quite-old model will suit *all* types of applications? shouldn’t the testing world consider welcoming new testing techniques?

Don’t get me wrong, in 2019 the testing pyramid, TDD and unit tests are still a powerful technique and are probably the best match for many applications. Only like any other model, despite its usefulness, it must be wrong sometimes. For example, consider an IOT application that ingests many events into a message-bus like Kafka/RabbitMQ, which then flow into some data-warehouse and are eventually queried by some analytics UI. Should we really spend 50% of our testing budget on writing unit tests for an application that is integration-centric and has almost no logic? As the diversity of application types increase (bots, crypto, Alexa-skills) greater are the chances to find scenarios where the testing pyramid is not the best match.

It’s time to enrich your testing portfolio and become familiar with more testing types (the next bullets suggest few ideas), mind models like the testing pyramid but also match testing types to real-world problems that you’re facing (‘Hey, our API is broken, let’s write consumer-driven contract testing!’), diversify your tests like an investor that build a portfolio based on risk analysis — assess where problems might arise and match some prevention measures to mitigate those potential risks

A word of caution: the TDD argument in the software world takes a typical false-dichotomy face, some preach to use it everywhere, others think it’s the devil. Everyone who speaks in absolutes is wrong :]

❎ Otherwise: You’re going to miss some tools with amazing ROI, some like Fuzz, lint, and mutation can provide value in 10 minutes

☺ Doing It Right Example: Cindy Sridharan suggests a rich testing portfolio in her amazing post ‘Testing Microservices — the sane way’

☺️Example: YouTube: “Beyond Unit Tests: 5 Shiny Node.JS Test Types (2018)” (Yoni Goldberg)

⚪️ 13. Component testing might be your best affair

✅ Do: Each unit test covers a tiny portion of the application and it’s expensive to cover the whole, whereas end-to-end testing easily covers a lot of ground but is flaky and slower, why not apply a balanced approach and write tests that are bigger than unit tests but smaller than end-to-end testing? Component testing is the unsung song of the testing world — they provide the best from both worlds: reasonable performance and a possibility to apply TDD patterns + realistic and great coverage.

Component tests focus on the Microservice ‘unit’, they work against the API, don’t mock anything which belongs to the Microservice itself (e.g. real DB, or at least the in-memory version of that DB) but stub anything that is external like calls to other Microservices. By doing so, we test what we deploy, approach the app from outwards to inwards and gain great confidence in a reasonable amount of time.

❎ Otherwise: You may spend long days on writing unit tests to find out that you got only 20% system coverage

☺ Doing It Right Example: Supertest allows approaching Express API in-process (fast and cover many layers)

⚪️ 14. Ensure new releases don’t break the API using consumer-driven contracts

✅ Do: So your Microservice has multiple clients, and you run multiple versions of the service for compatibility reasons (keeping everyone happy). Then you change some field and ‘boom!’, some important client who relies on this field is angry. This is the Catch-22 of the integration world: It’s very challenging for the server side to consider all the multiple client expectations — On the other hand, the clients can’t perform any testing because the server controls the release dates. Consumer-driven contracts and the framework PACT were born to formalize this process with a very disruptive approach — not the server defines the test plan of itself rather the client defines the tests of the… server! PACT can record the client expectation and put in a shared location, “broker”, so the server can pull the expectations and run on every build using PACT library to detect broken contracts — a client expectation that is not met. By doing so, all the server-client API mismatches are caught early during build/CI and might save you a great deal of frustration

❎ Otherwise: The alternatives are exhausting manual testing or deployment fear

☺ Doing It Right Example:

⚪️ 15. Test your middlewares in isolation

✅ Do: Many avoid Middleware testing because they represent a small portion of the system and require a live Express server. Both reasons are wrong — Middlewares are small but affect all or most of the requests and can be tested easily as pure functions that get {req,res} JS objects. To test a middleware function one should just invoke it and spy (using Sinon for example) on the interaction with the {req,res} objects to ensure the function performed the right action. The library node-mock-http takes it even further and factors the {req,res} objects along with spying on their behavior. For example, it can assert whether the http status that was set on the res object matches the expectation (See example below)

❎ Otherwise: A bug in Express middleware === a bug in all or most requests

☺ Doing It Right Example: Testing middleware in isolation without issuing network calls and waking-up the entire Express machine

⚪️ 16. Measure and refactor using static analysis tools

✅ Do: Using static analysis tools helps by giving objective ways to improve code quality and keep your code maintainable. You can add static analysis tools to your CI build to abort when it finds code smells. Its main selling points over plain linting are the ability to inspect quality in the context of multiple files (e.g. detect duplications), perform advanced analysis (e.g. code complexity) and follow the history and progress of code issues. Two examples of tools you can use are Sonarqube (2,600+ stars) and Code Climate (1,500+ stars)

Credit: Keith Holliday

❎ Otherwise: With poor code quality, bugs and performance will always be an issue that no shiny new library or state of the art features can fix

☺ Doing It Right Example: CodeClimat, a commercial tool that can identify complex methods:

⚪️ 17. Check your readiness for Node-related chaos

✅ Do: Weirdly, most software testings are about logic & data only, but some of the worst things that happen (and are really hard to mitigate ) are infrastructural issues. For example, did you ever test what happens when your process memory is overloaded, or when the server/process dies, or does your monitoring system realizes when the API becomes 50% slower?. To test and mitigate these type of bad things — Chaos engineering was born by Netflix. It aims to provide awareness, frameworks and tools for testing our app resiliency for chaotic issues. For example, one of its famous tools, the chaos monkey, randomly kills servers to ensure that our service can still serve users and not relying on a single server (there is also a Kubernetes version, kube-monkey, that kills pods). All these tools work on the hosting/platform level, but what if you wish to test and generate pure Node chaos like check how your Node process copes with uncaught errors, unhandled promise rejection, v8 memory overloaded with the max allowed of 1.7GB or whether your UX stays satisfactory when the event loop gets blocked often? to address this I’ve written, node-chaos (alpha) which provides all sort of Node-related chaotic acts

❎ Otherwise: No escape here, Murphy’s law will hit your production without mercy

☺Doing It Right Example: Node-chaos can generate all sort of Node.js pranks so you can test how resilience is your app to chaos

***Section 3️⃣: Measuring Test Effectiveness***

⚪️ 18. Get enough coverage for being confident, ~80% seems to be the lucky number

✅ Do: The purpose of testing is to get enough confidence for moving fast, obviously the more code is tested the more confident the team can be. Coverage is a measure of how many code lines (and branches, statements, etc) are being reached by the tests. So how much is enough? 10–30% is obviously too low to get any sense about the build correctness, on the other side 100% is very expensive and might shift your focus from the critical paths to the exotic corners of the code. The long answer is that it depends on many factors like the type of application — if you’re building the next generation of Airbus A380 than 100% is a must, for a cartoon pictures website 50% might be too much. Although most of the testing enthusiasts claim that the right coverage threshold is contextual, most of them also mention the number 80% as a thumb of a rule (Fowler: “in the upper 80s or 90s”) that presumably should satisfy most of the applications.

Implementation tips: You may want to configure your continuous integration (CI) to have a coverage threshold (Jest link) and stop a build that doesn’t stand to this standard (it’s also possible to configure threshold per component, see code example below). On top of this, consider detecting build coverage decrease (when a newly committed code has less coverage) — this will push developers raising or at least preserving the amount of tested code. All that said, coverage is only one measure, a quantitative based one, that is not enough to tell the robustness of your testing. And it can also be fooled as illustrated in the next bullets

❎ Otherwise: Confidence and numbers go hand in hand, without really knowing that you tested most of the system — there will also be some fear. and fear will slow you down

☺ Example: A typical coverage report

☺ Doing It Right Example: Setting up coverage per component (using Jest)

⚪️ 19. Inspect coverage reports to detect untested areas and other oddities

✅ Do: Some issues sneak just under the radar and are really hard to find using traditional tools. These are not really bugs but more of surprising application behavior that might have a severe impact. For example, often some code areas are never or rarely being invoked — you thought that the ‘PricingCalculator’ class is always setting the product price but it turns out it is actually never invoked although we have 10000 products in DB and many sales… Code coverage reports help you realize whether the application behaves the way you believe it does. Other than that, it can also highlight which types of code is not tested — being informed that 80% of the code is tested doesn’t tell whether the critical parts are covered. Generating reports is easy — just run your app in production or during testing with coverage tracking and then see colorful reports that highlight how frequent each code area is invoked. If you take your time to glimpse into this data — you might find some gotchas

❎ Otherwise: If you don’t know which parts of your code are left un-tested, you don’t know where the issues might come from

😕 Anti-Pattern Example: What’s wrong with this coverage report? based on a real-world scenario where we tracked our application usage in QA and find out interesting login patterns (Hint: the amount of login failures is non-proportional, something is clearly wrong. Finally it turned out that some frontend bug keeps hitting the backend login API)

⚪️ 20. Measure logical coverage using mutation testing

✅ Do: The Traditional Coverage metric often lies: It may show you 100% code coverage, but none of your functions, even not one, return the right response. How come? it simply measures over which lines of code the test visited, but it doesn’t check if the tests actually tested anything — asserted for the right response. Like someone who’s traveling for business and showing his passport stamps — this doesn’t prove any work done, only that he visited few airports and hotels.

Mutation-based testing is here to help by measuring the amount of code that was actually TESTED not just VISITED. Stryker is a JavaScript library for mutation testing and the implementation is really neat:

(1) it intentionally changes the code and “plants bugs”. For example the code newOrder.price===0 becomes newOrder.price!=0. This “bugs” are called mutations

(2) it runs the tests, if all succeed then we have a problem — the tests didn’t serve their purpose of discovering bugs, the mutations are so-called survived. If the tests failed, then great, the mutations were killed.

Knowing that all or most of the mutations were killed gives much higher confidence than traditional coverage and the setup time is similar

❎ Otherwise: You’ll be fooled to believe that 85% coverage means your test will detect bugs in 85% of your code

😕 Anti Pattern Example: 100% coverage, 0% testing

Doing It Right Example: Stryker reports, a tool for mutation testing, detects and counts the amount of code that is not tested (Mutations)

***Section 4️⃣ : CI & Other Quality Measures***

⚪️ 21. Enrich your linters and abort builds that have linting issues

✅ Do: Linters are a free lunch, with 5 min setup you get for free an auto-pilot guarding your code and catching significant issue as you type. Gone are the days where linting was about cosmetics (no semi-colons!). Nowadays, Linters can catch severe issues like errors that are not thrown correctly and losing information. On top of your basic set of rules (like ESLint standard or Airbnb style), consider including some specializing Linters like eslint-plugin-chai-expect that can discover tests without assertions, eslint-plugin-promise can discover promises with no resolve (your code will never continue), eslint-plugin-security which can discover eager regex expressions that might get used for DOS attacks, and eslint-plugin-you-dont-need-lodash-underscore is capable of alarming when the code uses utility library methods that are part of the V8 core methods like Lodash._map(…)

❎ Otherwise: Consider a rainy day where your production keeps crashing but the logs don’t display the error stack trace. What happened? Your code mistakenly threw a non-error object and the stack trace was lost, a good reason for banging your head against a brick wall. A 5min linter setup could detect this TYPO and save your day

😕 Anti Pattern Example: The wrong Error object is thrown mistakenly, no stack-trace will appear for this error. Luckily, ESLint catches the next production bug

⚪️ 22. Shorten the feedback loop with local developer-CI

✅ Do: Using a CI with shiny quality inspections like testing, linting, vulnerabilities check, etc? Help developers run this pipeline also locally to solicit instant feedback and shorten the feedback loop. Why? an efficient testing process constitutes many and iterative loops: (1) try-outs -> (2) feedback -> (3) refactor. The faster the feedback is, the more improvement iterations a developer can perform per-module and perfect the results. On the flip, when the feedback is late to come fewer improvement iterations could be packed into a single day, the team might already move forward to another topic/task/module and might not be up for refining that module.

Practically, some CI vendors (Example: CircleCI load CLI) allow running the pipeline locally. Some commercial tools like wallaby provide highly-valuable & testing insights as a developer prototype (no affiliation). Alternatively, you may just add npm script to package.json that runs all the quality commands (e.g. test, lint, vulnerabilities) — use tools like concurrently for parallelization and non-zero exit code if one of the tools failed. Now the developer should just invoke one command — e.g. ‘npm run quality’ — to get instant feedback. Consider also aborting a commit if the quality check failed using a githook (husky can help)

❎ Otherwise: When the quality results arrive the day after the code, testing doesn’t become a fluent part of development rather an after the fact formal artifact

☺ Doing It Right Example: npm scripts that perform code quality inspection, all are run in parallel on demand or when a developer is trying to push new code

⚪️ 23. Perform e2e testing over a true production-mirror

✅ Do: End to end (e2e) testing are the main challenge of every CI pipeline — creating an identical ephemeral production mirror on the fly with all the related cloud services can be tedious and expensive. Finding the best compromise is your game: Docker-compose allows crafting isolated dockerized environment with identical containers using a single plain text file but the backing technology (e.g. networking, deployment model) is different from real-world productions. You may combine it with ‘AWS Local’ to work with a stub of the real AWS services. If you went serverless multiple frameworks like serverless and AWS SAM allows the local invocation of Faas code.

The huge Kubernetes eco-system is yet to formalize a standard convenient tool for local and CI-mirroring though many new tools are launched frequently. One approach is running a ‘minimized-Kubernetes’ using tools like Minikube and MicroK8s which resemble the real thing only come with less overhead. Another approach is testing over a remote ‘real-Kubernetes’, some CI providers (e.g. Codefresh) has native integration with Kubernetes environment and make it easy to run the CI pipeline over the real thing, others allow custom scripting against a remote Kubernetes.

❎ Otherwise: Using different technologies for production and testing demands maintaining two deployment models and keeps the developers and the ops team separated

☺️ Example: a CI pipeline that generates Kubernetes cluster on the fly (Credit: Dynamic-environments Kubernetes)

deploy:
stage: deploy
image: registry.gitlab.com/gitlab-examples/kubernetes-deploy
script:
- ./configureCluster.sh $KUBE_CA_PEM_FILE $KUBE_URL $KUBE_TOKEN
- kubectl create ns $NAMESPACE
- kubectl create secret -n $NAMESPACE docker-registry gitlab-registry --docker-server="$CI_REGISTRY" --docker-username="$CI_REGISTRY_USER" --docker-password="$CI_REGISTRY_PASSWORD" --docker-email="$GITLAB_USER_EMAIL"
- mkdir .generated
- echo "$CI_BUILD_REF_NAME-$CI_BUILD_REF"
- sed -e "s/TAG/$CI_BUILD_REF_NAME-$CI_BUILD_REF/g" templates/deals.yaml | tee ".generated/deals.yaml"
- kubectl apply --namespace $NAMESPACE -f .generated/deals.yaml
- kubectl apply --namespace $NAMESPACE -f templates/my-sock-shop.yaml
environment:
name: test-for-ci

⚪️ 24. Parallelize test execution

✅ Do: When done right, testing is your 24/7 friend providing almost instant feedback. In practice, executing 500 CPU-bounded unit test on a single thread can take too long. Luckily, modern test runners and CI platforms (like Jest, AVA and Mocha extensions) can parallelize the test into multiple processes and achieve significant improvement in feedback time. Some CI vendors do also parallelize tests across containers (!) which shortens the feedback loop even further. Whether locally over multiple processes, or over some cloud CLI using multiple machines — parallelizing demand keeping the tests autonomous as each might run on different processes

❎ Otherwise: Getting test results 1 hour long after pushing new code, as you already code the next features, is a great recipe for making testing less relevant

☺️ Doing It Right Example: Mocha parallel & Jest easily outrun the traditional Mocha thanks to testing parallelization (Credit: JavaScript Test-Runners Benchmark)

⚪️ 25. Stay away from legal issues using license and plagiarism check

✅ Do: Licensing and plagiarism issues are probably not your main concern right now, but why not tick this box as well in 10 minutes? A bunch of npm packages like license check and plagiarism check (commercial with free plan) can be easily baked into your CI pipeline and inspect for sorrows like dependencies with restrictive licenses or code that was copy-pasted from Stackoverflow and apparently violates some copyrights

❎ Otherwise: Unintentionally, developers might use packages with inappropriate licenses or copy-paste commercial code and run into legal issues

☺️ Doing It Right Example:

//install license-checker in your CI environment or also locally
npm install -g license-checker

//ask it to scan all licenses and fail with exit code other than 0 if it found unauthorized license. The CI system should catch this failure and stop the build
license-checker --summary --failOn BSD

⚪️26. Constantly inspect for vulnerable dependencies

✅ Do: Even the most reputable dependencies such as Express have known vulnerabilities. This can get easily tamed using community tools such as npm audit, or commercial tools like snyk (offer also a free community version). Both can be invoked from your CI on every build

❎ Otherwise: Keeping your code clean from vulnerabilities without dedicated tools will require to constantly follow online publications about new threats. Quite tedious

☺️ Example: NPM Audit results

⚪️ 27. Automate dependency updates

✅ Do: Yarn and npm latest introduction of package-lock.json introduced a serious challenge (the road to hell is paved with good intentions) — by default now, packages are no longer getting updates. Even a team running many fresh deployments with ‘npm install’ & ‘npm update’ won’t get any new updates. This leads to subpar dependant packages versions at best or to vulnerable code at worst. Teams now rely on developers goodwill and memory to manually update the package.json or use tools like ncu manually. A more reliable way could be to automate the process of getting the most reliable dependency versions, though there are no silver bullet solutions yet there are two possible automation roads: (1) CI can fail builds that have obsolete dependencies — using tools like ‘npm outdated’ or ‘npm-check-updates (ncu)’ . Doing so will enforce developers to update dependencies. (2) Use commercial tools that scan the code and automatically send pull requests with updated dependencies. One interesting question remaining is what should be the dependency update policy — updating on every patch generates too many overhead, updating right when a major is released might point to an unstable version (many packages found vulnerable on the very first days after being released, see the eslint-scope incident). An efficient update policy may allow some ‘vesting period’ — let the code lag behind the @latest for some time and versions before considering the local copy as obsolete (e.g. local version is 1.3.1 and repository version is 1.3.8)

❎ Otherwise: Your production will run packages that have been explicitly tagged by their author as risky

☺️ Example: ncu can be used manually or within a CI pipeline to detect to which extent the code lag behind the latest versions

⚪️ 28. Other, non-Node related, CI tips

✅ Do: This post is focused on testing advice that is related to, or at least can be exemplified with Node JS. This bullet, however, groups few non-Node related tips that are well-known

1. Use a declarative syntax. This is the only option for most vendors but older versions of Jenkins allows using code or UI
2. Opt for a vendor that has native Docker support
3. Fail early, run your fastest tests first. Create a ‘Smoke testing’ step/milestone that groups multiple fast inspections (e.g. linting, unit tests) and provide snappy feedback to the code committer
4. Make it easy to skim-through all build artifacts including test reports, coverage reports, mutation reports, logs, etc
5. Create multiple pipelines/jobs for each event, reuse steps between them. For example, configure a job for feature branch commits and a different one for master PR. Let each reuse logic using shared steps (most vendors provide some mechanism for code reuse
6. Never embed secrets in a job declaration, grab them from a secret store or from the job’s configuration
7. Explicitly bump version in a release build or at least ensure the developer did so
8. Build only once and perform all the inspections over the single build artifact (e.g. Docker image)
9. Test in an ephemeral environment that doesn’t drift state between builds. Caching node_modules might be the only exception

❎ Otherwise: You‘ll miss years of wisdom

⚪️ 29. Build matrix: Run the same CI steps using multiple Node versions

✅ Do: Quality checking is about serendipity, the more ground you cover the luckier you get in detecting issues early. When developing reusable packages or running a multi-customer production with various configuration and Node versions, the CI must run the pipeline of tests over all the permutations of configurations. For example, assuming we use mySQL for some customers and Postgres for others — some CI vendors support a feature called ‘Matrix’ which allow running the suit of testing against all permutations of mySQL, Postgres and multiple Node version like 8, 9 and 10. This is done using configuration only without any additional effort (assuming you have testing or any other quality checks). Other CIs who doesn’t support Matrix might have extensions or tweaks to allow that

❎ Otherwise: So after doing all that hard work of writing testing are we going to let bugs sneak in only because of configuration issues?

☺ Example: Using Travis (CI vendor) build definition to run the same test over multiple Node versions

language: node_js
node_js:
  - "7"
  - "6"
  - "5"
  - "4"
install:
  - npm install
script:
  - npm run test

Thank You. Other articles you might like

• 📗 Want to take your testing skills to the extreme? Consider visiting my comprehensive course ‘Testing Node.js & JavaScript From A To Z’
• Checklist: Node.js production best practices (August 2018)
• 19 ways to become a better Node.js developer in 2019
• Node.js security best practices (September 2018)
• YouTube: 5 advanced and shiny testing techniques
• Node.js best practices — 79 best practices for a robust Node application

⭐ Want more? follow me on Twitter⭐

✏️ Have your own testing tip? PR here and I’ll be sure to update this article ✏️

Short Intro

I’ve compiled 19 skills and topics you might find valuable in 2019. Please don’t get overwhelmed — neither I nor most other developers are familiar with every single topic. These are just the exciting things on my radar, and the JavaScript horizon is never-ending.

My name is Yoni Goldberg, an independent Node.JS consultant and the co-author of Node.js best practices. I work with customers at the USA, Europe and Israel on polishing their Node.js applications. Among my services are code, application and architecture review, testing & CI, advanced training sessions, and others which you may find here. Follow me on Twitter.

📗 Want to take your testing skills to the extreme? Consider visiting my comprehensive course ‘Testing Node.js & JavaScript From A To Z’.

Reviewed & improved by Bruno Scheufler and the Reddit community (special thanks to @pysouth, @fullheap, @ReginaldBull and @relativityboy) and.

1. Add some types and schemas. Typescript is a great candidate in 2019

Coding in a typeless manner has proven to be counter-productive and error-prone (see research). That doesn’t mean you must go all the way toward strictly-typed syntax, rather choose your desired degree of schematic code either by just validating your entities/models using JSON schema (or Joi), to annotating your vanilla JS with static typing (see Facebook flow) or go all the way to using almost fully-typed syntax using Typescript. The later has gained remarkable momentum in 2018 and seems to find its way into the Node land consensus. If you do plan on using Typescript, ask yourself if your usage should span beyond just Typing features, otherwise using interfaces and abstract classes takes you into a paradigm you never meant to try

Examples:

• Explicit models schemas with JSON Schema
• Static typing over vanilla JS using Facebook flow
• Typed syntax with Typescript

2. Enrich your Linters

Linters are a free lunch, with 5 min setup you get for free an auto-pilot guarding your code and catching significant issue as you type. Gone are the days where linting was about cosmetics (no semi-colons!). Nowadays, Linters can catch severe issues like errors that are not thrown correctly and losing information, promises that are never resolved and other sorrows that you never really meant to include in your code

Examples:

• eslint-plugin-chai-expect can discover tests without assertions
• eslint-plugin-promise can discover promises with no resolve (your code will never continue)
• eslint-plugin-security can discover eager regex expressions that might get used for DOS attacks

3. Be a bit more Java and a byte less Ruby — deepen your architecture knowledge

Very little architecture and design knowledge are spread around the NodeJS eco-system, everyone speaks about Microservices but only a few about their internal structure. As a result, most applications and examples are MVC concepts and other doubtful patterns from Ruby. What’s wrong with that? MVC, for example, was build for serving content and is an underwhelming technique for structuring a robust backend (Uncle Bob: “MVC is a delivery mechanism, not an application architecture”). Can you really describe your entire business logic, rules, data access, communication with other Microservices in two classes — Controller & Model? See examples below for other design issues and a potential remedy

I absolutely don’t recommend embracing heavy Java/Spring patterns (we came here to Node-Land for a reason, didn’t we? :)), just cherry-pick few ideas that provide great value without sacrificing the app simplicity

Examples:

• Did you read my Node.js best practices section 1 — architecture?
• Avoid cluttering your business logic with Express objects, read about Domain-Driven Design (see a shortened version of this Novel book here) & Hexagonal architecture
• Mixing logic & data-access code in a single class (Active Record pattern — very popular among developers that use Mongoose and Sequelize) easily leads to bloated objects that are harder to test. Consider using the data mapper pattern instead
• Skim through the code of this nice Node.js boilerplate that implements Domain Driven Design and clean architecture

4. Plan how to utilize Async-Hooks to reach better tracing and context

That single threaded model has one major drawback — requests lose context: as they flow through multiple files and performs asynchronous operations, variables are not preserved throughout their life-cycle. Why is this painful? For example, often developers wish to include within each log entry a unique identifier so later one can correlate all the log entries of the same request — this is not very easy in 2018. 2019 brings new shiny new stuff and async-hooks is one of them (not entirely new but should get out of experimental mode soon )— simply put, it’s a mechanism to inject custom code anytime some async operations are starting and ending. Given this, it’s possible to correlate all the code of the same request and preserve the context. This lays the foundation for many custom packages that takes Node’s tracing and context to the next level

Examples:

• cls-hooked allows sharing variables and context through the entire request life-cycle
• Jaeger client will visualize the entire request flow through the system, even across Microservices and servers (part of the open-tracing standard and requires a dedicated server to record all the activities)
• Learn about async-hooks opportunities and how to code against it. By @guysegev

5. Understand the latest ‘Serverless’ features: It’s now ready to battle on the robust infrastructure field (Kubernetes killer?)

Note: The words FaaS and Serverless are used here interchangeably though they are not exactly the same thing. Practically I’m referring to cloud vendors FaaS services like Lambda and Google Functions

Originally FaaS was meant for development micro-tasks and not for robust “Microservice” applications. As their popularity grew, so the appetite of the cloud vendors, soon new features piled-up and suddenly now in 2019 it seems like a worthy infrastructure for robust applications. Can it now compete with Kubernetes and serve large applications? Some see Serverless and Faas as orthogonal technologies but practically every new cloud application in 2019 will have to choose (literally this choice is shown on each cloud vendor UI) one of the three: (1) bare-metal instance like EC2 or GCP compute (2) Kubernetes or (3) FaaS. Consequently, being able to compare K8S vs FaaS/serverless and telling the consequences becomes a mandatory design skill.

p.s. The examples below relate to AWS for convenience only.

Examples:

• AWS Lambda SAM tool allows defining and running FaaS locally
• AWS Lambda now supports canary deployments!
• AWS Lambda layers allow reusing logic among multiple FaaS (simulating the domain/business logic layer of a typical Microservice)

6. Meet the latest JavaScript features that are turning green soon

I’m not a big fan of chasing for every new language feature, sometimes these shiny toys work against the code simplicity and clarify. From time to time some really valuable Javascript features are presented (like the introduction of async/await two years ago) so it’s worthwhile watching the TC39 proposal list and node.green to identify attractive new features that fit your coding style

Examples:

• Class level field is at stage 3 (last phase) and might turn green in 2019
• BigInt is at stage 3 (last phase) and might help when interacting with other microservices, machines or data-warehouse that produce gigantic numbers
• Async iterators (Matt Krick) and †promise-finally are already green and worth checking out if you missed one of those

7. Become intimately familiar with at least one more API technology. GraphQL is a great candidate in 2019

REST-style API is great for the purpose it was built for: having great control over entities modifications and queries. Have a finance records system? you probably want to design very strict endpoints that a single and explicit data model. REST-style, however, does fall short in other very common use cases like executing similar queries that might return a different set of data, a low-bandwidth network that dictates minimizing the API payload, machine-to-machine communications that emphasize speed, to name of a few. Should you switch? Absolutely not, just mix. APIs are not your architecture, they are just a port (i.e. entry point) to your application and multiple API-styles can easily co-exist. Even on top of a single web framework like Express.

So which one to learn? Your best bet is probably GraphQL which is carried over a great momentum straight into the main-stream. Its echo-system has greatly matured and it serves very popular use cases like dynamic search and hierarchic data source. Grpc, on the other hand, is still a niche technology that fits well server to server communication use cases where minimum overhead is appreciated (e.g. Pub-sub/message-queue system).

Examples:

• Learning by contrasting is great — REST vs Graph vs grpc
• Skim through a GraphQL, Node & Express tutorial
• Watch a short YouTube (11 min) — what is GraphQL?

8. Go beyond unit & integration tests — enrich your testing portfolio with shiny new testing techniques

Already familiar with the testing pyramid, unit, integration and end-to-end tests? great, these are the foundation for a successful testing strategy. However, in the past 10 years, the development world had gone through dramatic changes but the testing models were left intact leaving us wondering how to test things like Microservice, rich frontends, and Serverless. Some modern techniques compliment the traditional stacks and sometimes can even replace it to achieve a leaner testing strategy with a better ROI

Examples:

• Consumer-driven contracts can prevent breaking API issues between Microservices or between you and the consumer of your API
• Snapshot tests can be used not only for UI but also to prevent API regression
• Component tests are a balanced approach for Microservice testing
• See my YouTube video on “Beyond Unit Tests: 5 Shiny Node.JS Test Types (2018)”

9. Align your monitoring with SRE/DevOps best practices

In 2019, even a medium-sized app might constitute dozens of physical moving parts and staying on top of this big band orchestra should be done with a great care. Yet, most developers hadn’t taken their time to learn the monitoring & alerting lessons that site reliability engineers are willing to teach. As an example, often developers prioritize and focus on internal hardware metrics like CPU and memory instead of starting from metrics that directly and undeniably affects the end-user immediately like error rate or latency (“I call this symptom­-based monitoring”, from ‘My philosophy on alerting’). Those customer-facing metrics are sometimes referred to as ‘The golden signals’ and in 2019 you might wanna start there and embrace similar best practices

• Read about the 4 monitoring ‘Golden signals’
• Read Google Site Reliability Engineering book, or at least the chapter on monitoring
• The package request-stats might help to extract those customer-facing metrics so you can share with your monitoring system

10. Think like an attacker: increase the security level by learning attack tools and techniques

If you can’t think like an attacker, you can’t think like a defender. In 2019, you shouldn’t outsource the defense work to third-party companies or rely solely on static security scanners: The amount of attack types is overwhelming (The development pipeline and NPM are the latest trends), the pace of application change is untamable — two days after conducting a security workshop the team can add several new AWS services, database types and new IAM role for example… So implicitly developers become the biggest threat and educating them seems like the ultimate remedy. You have to bake security DNA into yourself and your team and add a security touch to everything.

Once you start doing this, it turns out to be not that scary. Just become familiar with common attack types and tools, draw your application architecture and flows and think how you would attack this. With time, unconsciously you’ll start mind security in every design decision and every code line

Examples:

• Try OWASP ZAP — a rich assessment & penetration tool that allows even newbies explore the security level of application
• Read my list of Node.js Security best practices which contains 23+ attack ideas including JavaScript code examples
• Conduct a monthly threats analysis meeting where the team tries to look at the application design and propose attacks. sound boring? not necessarily if you add some gamification and reward members that find an exploit, or run a competition between a blue team that designs a module vs the read team which tries to find exploits

11. Have a package update strategy. A lesson learned in 2018: updating too soon is a dangerous practice

Teams usually hold of one of the two npm/Yarn package update strategy: (1) update as soon as possible, sometimes even using an automated process (2) no update strategy at all, sometimes someone updates based on goodwill. While the 1st approach seems superior, surprisingly it turned out to be the riskiest approach in 2018: all the malicious packages incidents like flat-stream were discovered by the community within 40 days, those who waited and didn’t update too soon were saved. Consider formalizing the update strategy using automated tools and find the sweet spot between not updating at all to updating too soon

Examples:

• npq by Liran Tal is a great advisory package installer that minds also the release date
• Commercial tools like greenkeeper will PR once a package is updated. Unfortunately, none is still capable of suspending the update until a release is proven safe

12. Perform gradual installations, separate between the deploy and release phases

In 2019 you might find it useful to perform safer deployments that are not rolled-out in a single shot and leaving your heart beating like a hammer. On the safer side, granular deployment (a.k.a canary) suggests to separate into 3 phases: (1) Deployment — send a new code to an isolated and new production area (e.g. a new Kubernetes service or a new machine instance). At this stage, it serves nobody so no fear attached (2) Test — few people can now work against and test the new code in its most realistic environment, the production (3) Release — gradually allow more users to hit the new version (e.g. the entire east coast) and once you feel enough confident enough you can fade out the older version

A word of caution: performing full-blown canary deployments in 2019 is still very expensive as it requires to orchestrate many infrastructure parts like routing, and monitoring. Therefore, consider starting with simple and semi-manual canary deployments (e.g. manually spin-up more machines with the new version based on the monitoring metrics)

Examples:

• Learn more about Canary releases
• If you’re willing to go all the way to fancy canary deployments, Spinnaker is a robust deployment platform to look at

13. Kubernetes ate the world

An obvious trend, shortened item: Kubernetes (K8S), an infrastructure for application components that seamlessly provide networking, scale-out, deployment and other backbone services, is now almost a de-facto standard for hosting applications. Its popularity is phenomenal: supported by all cloud vendors, has an unmatched echo system of extensions, even 54% of enterprises already have at least one K8S cluster. If you’re a beginner, this link provides a nice hands-on overview. Already did your first K8S steps? make sure to be familiar with Istio, K-Native, Kuberenes jobs, the internals overview, network policies, Helm, Scaffold. Bottom line: Any hour you spent on deepening your K8S skills will pay-off

14. Blockchain technologies embody some great opportunities

An obvious trend, shortened item: Not only for Bitcoin and crypto functionality but blockchains can also be used for any distributed system handling transactions.

15. Gain solid machine learning skills, or at least speak intelligently about it

An obvious trend, shortened item: Not much to add here, maybe the leading trend of our times. Unfortunately, I’m clueless about machine library and my goal for 2019 is at least becoming comfortable to intelligently speak about it and identify quick-win opportunities (e.g. JS libraries like tensorflow.js and brain.js that can provide some insights without robust infrastructure)

16. Skim through the code of selected open source libraries

An obvious trend, shortened item: Beware, working on the same project for a long time using the same set of technologies might narrow your vision and hide great alternatives. Strive to investigate other projects frequently, mostly successful open source projects.

17. Deepen your Linux OS understanding, focus on the anatomy of a Linux process

Shortened item: understanding the Linux process will gain you a true competitive advantage as it influences many development tasks like monitoring, guarding processes (e.g. restarting), working with Docker, shutting down gracefully and many others. Strive to understand the life-cycle of a process, signals, the permission model, common commands, types of processes and more. This tutorial covers most of the basics

18. Dive deeper into the Node.js internals

I really like Ryan Dahl’s (creator of Node.js v0,1) quote: “You can never understand everything. But, you should push yourself to understand the system”. Gaining a solid understanding of the underlying machine can be proven as valuable when need to handle hairy production issues or designing some infrastructural component (e.g. monitor the event loop performance). You might already be familiar with the core building blocks like v8 and libuv. Isn’t 2019 a great time to deepen your journey within the Node.js rabbit hole and learn, for example, exactly what is happening inside each libuv event loop cycle? or maybe learn how the interaction with the operating system IO is done (E.g. active handles)?

Examples:

• Understand what’s happening within each event loop cycle by Deepal Jayasekara
• Learn how to pack C/C++ code as NPM module by Konstantin Tarkus
• Visit this great in-depth post that covers the entire Node.js internals by Eugene Obrezkov

19. Last but not least: learn using scientific methods

What you learn and internalize will shape your future career. Nevertheless, many developers neither have a learning strategy nor do they learn how to learn effectively using scientific methods. Consider a meeting about “Preventing JavaScript type bugs”, the VP asks to keep using vanilla JavaScript without refactoring the entire codebase, (no Typescript…), suddenly your colleague suggests using Facebook flow, everybody in the room likes it! You suddenly remember that you read about it once but it was never internalized and just slipped through your mind. How come? apparently, there is phenomena named ‘illusion of competness’ which explains why we forget things: You may spend precious 1 hour on reading a blog post but you’re fooling yourself and won’t remember a thing within a few days! Studies showed that if you try to speak about it later with someone or just read again a summary the next day you can greatly improve the chances of memorizing this concept. There are various other techniques that will help you remember and pull the right piece of knowledge at the right time (see examples below), spending a few hours on learning how to learn can make a great deal for your career in 2019!

Examples:

• Take the amazing course ‘Learning How to Learn’
• Use chunking and categorization while learning
• Reading about some new technology? Compare it to existing things you’re familiar with, speak about it with your teammates, ask yourself the “so what” question — why is it needed, draw a diagram, re-read a summary and help your brain internalize and categorize it so you’ll be able to fetch it in the important situation!

Thank You. Other articles you might like

• 📗 Liked the content here and want to get 10+ hours course on Node.js quality and testing? Visit my online course ‘Testing Node.js & JavaScript From A To Z’
• Checklist: Node.js production best practices (August 2018)
• Node.js security best practices (September 2018)
• YouTube: 5 advanced and shiny testing techniques
• Node.js best practices — 77 best practices for a robust Node application

⭐ Want more? follow me on Twitter⭐