Stories by anurag sharma on Medium

Best Engineering Blogs/Articles/Videos for System Design

anurag sharma — Thu, 04 Apr 2024 16:57:56 GMT

Following Engineering Blogs is one of the best ways to understand how the engineering teams at top tech companies function and how they build scalable systems.

It also helps in preparing for System Design Interviews as you get to understand the trade-offs that these companies took in making their architecture scalable based on their requirements.

I have curated a list of the top engineering blogs and have also added the recommended articles from each of them to help you learn more about their architecture and prepare for System Design (HLD) Interviews.

I would highly suggest you go through 1–2 articles/videos every day. It would hardly take 30 mins/day. Consistency is key.

You do not need to understand everything completely. Just a brief idea about what’s happening is good enough. Things will start making sense once you’ve read many articles. If you find something interesting and want to learn more, then Google for more information.

Netflix TechBlog

Link: Netflix TechBlog

Netflix has one of the best tech blogs in the industry. They have written a lot about their learnings and practices across a bunch of different areas. Their blog and open source infra have helped a lot of companies scale and adopt Netflix’s engineering practices. They have also been pioneers in the adoption of microservices architecture across the industry.

Recommended Links:

Overview

Microservices

Fault Tolerance and Resiliency

Chaos Engineering

Streaming

Architecuture of other products

Common Infrastructure

Recommendation Engine

Uber Engineering Blog

Link: Uber Engineering Blog

Recommended Links:

Airbnb Engineering Blog

Link: Infrastructure at Airbnb

Recommended Links:

Hotstar Tech Blog

Hotstar has one of the top tech blogs in India.

Link: Hotstar Tech Blog

Recommended Links:

Myntra Engineering Blog

Myntra has been at the forefront of innovation in the e-commerce space in India. Their engineering blog provides a lot of insights regarding the same.

Link: Myntra Engineering

Recommended Links:

Swiggy Tech Blog

Link: Swiggy Bytes

Recommended Links:

Flipkart Tech Blog

Link: Flipkart Tech Blog

Recommended Links:

Gojek Tech Blog

Link: Gojek Tech Blog

Grab Tech Blog

Link: Grab Tech

Recommended Links:

Discord Engineering Blog

Link: Discord Engineering Blog

Recommended Links:

Twitter Engineering Blog

Recommended Links:

Instagram Engineering Blog

Link: Instagram Engineering

Recommended Links:

Facebook, Whatsapp and YouTube

Recommended Links:

LinkedIn Engineering Blog

Link: LinkedIn Engineering Blog

Recommended Links:

Pinterest Engineering Blog

Link: Pinterest Engineering Blog

Recommended Links:

Dropbox Engineering Blog

Link: Infrastructure — Dropbox

Recommended Links:

Stripe Engineering Blog

Link: Stripe Engineering Blog

Recommended Links:

Other Engineering Blog

Multi-tenancy with spring boot

I hope that you found this article to be extremely insightful. If you feel that I’ve missed some blog/article/video then do ping me.

Understanding Java Optional

anurag sharma — Sat, 19 Dec 2020 10:26:36 GMT

Photo by Arnel Hasanovic on Unsplash

Optional introduced in Java in version 8 is often misused. Eliminating nulls isn’t a goal of Optional.

The primary design goal for Optional is to be used as the library method return value of functions when a return value might be absent. The intent is that the caller immediately checks the Optional and extract the actual value if it is present. If the value is absent, the caller can substitute a default value, throw an exception, or apply some other policy. This is typically done by chaining fluent method calls off the end of a stream pipeline (or other methods) that return Optional values.

Let us discuss where not to use optional:

Places Not to Use Optional

Avoid using Optional in method parameters

With an Optional parameter, you go from having 2 possible inputs: null and not-null, to three: null, non-null-without-value, and non-null-with-value. Add to that the fact that overloading has long been available to convey that some parameters are optional, and there's really no reason to have Optional parameters. Code analysis tools such as SonarQube and IntelliJ flag Optional in method parameters as a major code smell(RSPEC-3553).

Since the main point of Optional is to provide a means for a function returning a value to indicate the absence of a return value. See this discussion. This allows the caller to continue a chain of fluent method calls.

Passing an optional argument to a method, this could be made to work, but it’s rather clumsy. Suppose you have a method that takes a string followed by an optional second string. Accepting an Optional as the second arg would result in code like this:

foo("bar", Optional.of("baz"));
foo("bar", Optional.empty());

Even accepting null is nicer:

foo("bar", "baz");
foo("bar", null);

Probably the best is to have an overloaded method that accepts a single string argument and provides a default for the second:

foo("bar", "baz");
foo("bar");

This does have limitations, but it’s much nicer than either of the above.

Avoid using Optional in fields

Java Optional is also not recommended to be used as fields in POJOs. Java Optional is not serializable and if you really have a requirement of the optional field you can use nullable fields with getter returning Optional.

Avoid using Optional in collections

Sometimes people want to put Optional into collections, like List> or Map>. This too is usually a bad idea. This may seem required to distinguish between the absence of the record and null record. It's often better to replace these usages of Optional with Null-Object values (not actual null references), or simply to omit these entries from the collection entirely.

Memory Footprint

Optional is a box and consumes extra 16 bytes since it also adds one level of reference redirection it potentially adds to GC pressure and cache misses.

Adapting Between Null and Optional

Sometimes you need to adapt Optional-using code to code that wants null,
or vice-versa

• If you have a nullable reference and you need an Optional

Optional opt = Optional.ofNullable(ref)

• If you have an Optional and you need a nullable reference

opt.orElse(null)

Optional is not serializable

Even though Guava Optional is serializable, Java optional is not serializable by design.

Usage at Whatfix

At Whatfix, we use Optional as the library method return value of functions when a return value might be absent.

Understanding Java Optional was originally published in WhatfixEngineeringBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building a Generic Enterprise Application Integration Pipeline

anurag sharma — Wed, 08 Apr 2020 18:46:22 GMT

One pipeline to rule them all

At Whatfix, one of our solutions is to show context-aware and personalized help as an overlay on enterprise applications. Integrations with enterprise applications allow us to efficiently contextualize content for our users. Based on these integrations we are able to target information based on:

User roles
Survey responses
User cohorts, etc.

In this post, we will discuss the high-level design and the mental-model of our generic integration pipeline to connect to enterprise applications to support a variety of use cases including paginated APIs and asynchronous APIs etc.

Integration Pipeline and Steps

Simple integration pipeline

Each integration pipeline can have multiple steps. Integration steps are of two types:

REST Connector: to call REST APIs of a system. Typical configurations for this step are URL to call, the HTTP request method, authentication details, request payload etc.
Transformation: to convert the response object of one generic REST connector into the request object of another generic REST connector using JavaScript.

Complexities of the Real World

There are many use cases which may not be as simple as the three-step pipeline and require more steps.

1. Pagination

Real-world APIs may be paginated. Pagination itself can be of many types, such as:

Cursor based
Offset based

For more details of the pagination, you can read the evolution of pagination at slack.

2. Asynchronous APIs

Systems with a large amount of data may optimize storage and compute by exposing APIs that may not return data instantaneously but the response contains a URL to poll for the status of the background job to fetch data. Once the job is complete, the job status call response contains the URL to be finally called to get data.

3. APIs returning links to be called further

Certain APIs return a list of URL links to be further called to get required data. In such cases, we need to iterate over the list of URLs and make calls to get the required data.

Supporting Pagination, Async APIs and URL links in the Integration Pipeline

To satisfy a wide variety of use cases with similar UX as of simple generic pipeline, we turbocharged our transformation step of the pipeline.

We defined two keywords nextLinks (a JavaScript array of objects) and data (a JavaScript object), which could be set in the transformation step.

We pick the nextLinks and run the previous REST Connector step again by taking the nextLinks as URL parameters. And data is always passed down further steps for processing.

Turbocharged transformation step

The condition to break looping from the transformation step would be when nextLinks is null.

For Async APIs, the transformation step could look as follows:

Next Steps

Now that we are done with building a generic integration pipeline we are working to build specific connectors for different enterprise applications.

This is the work of the Data Integration team at Whatfix. Thanks to the Data Integration team and Nibu Thomas for the review.

Building a Generic Enterprise Application Integration Pipeline was originally published in WhatfixEngineeringBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.