The infamous question among beginners and programming enthusiasts who are just getting started is-

“Should I know the topics of computer science and mathematics to become a good software developer or engineer or programmer?”

If God can make a random person billionaire every time this question was asked, then the price of a single bread slice would be 1 Billion 💸💸💸

If you are just starting out and search in Google whether math is really necessary then there is a 99 in 100 chance that you come across these type of answers

JavaScript is not available.

Once headmaster (HM) of my school asked me: who do you want to become?Me: a programmer.HM: are you good at math?Me: no.HM: oh, you'll need that in order to become a programmer.Me: ...Now, 8 years later, I am a Web developer, working from home.I'm still not good at math.

Once headmaster (HM) of my school asked me: who do you want to become?

Me: a programmer.

HM: are you good at math?

Me: no.

HM: oh, you’ll need that in order to become a programmer.

Me: …

Now, 8 years later, I am a Web developer, working from home.

I’m still not good at math.

The problem with these types of answers is, although they are not wrong but often taken out of context and quite misleading for beginners who fail to distinguish between developers and engineers.

So, let’s answer some basic questions before we go for a debate over the importance of Math.

In the software industry, the terms developer, engineer and programmer are used interchangeably.

Who is a Software Developer?

A Software Developer is a person who develops applications or software using libraries, APIs, and frameworks.

The libraries, APIs, and frameworks do all the heavy-lifting and the job of the developer is to piece everything together to meet the requirements.

The developer needs to have good logical thinking to reason the components and working of the software. Developers are not expected to solve problems. The job of a software developer is to convert requirements into working code.

When a person is being interviewed for a developer position, the organization expects the person to be fluent in the programming languages, libraries, and frameworks that are being used by them in the production.

For example, if you need to develop a website with animations, you just use the framework. Usually, animation requires the knowledge of mathematics but the framework does all the heavy and exposes an API.

Another example, every database is built on top of complex algorithms, data structures, and memory management techniques, etc., which are used by software developers.

Who is a Software Engineer?

A software engineer is a person who develops software, frameworks, and also systems that will be used by other developers.

These people are the ones who do all the heavy work that’s going into libraries and frameworks etc.

Like any other engineer, Software Engineer should have strong problem-solving, and math skills. Additionally, knowledge of algorithms, data structures is a must no matter what you hear on public forums.

Apart from having those skills, they are also expected to have minimum knowledge about Operating Systems, Databases, Networks, Distributed Computing, Architectures, and Design Patterns, etc.

When a person is being interviewed for an engineer position, the organization expects the person to be fluent in the algorithms, data structures, and minimum knowledge in Operating Systems, Databases, Networks, Distributed Computing, Architectures, and Design Patterns, etc.

This is the reason why many companies during the whiteboard interviews, don’t insist on a particular language. Even in the job postings, they just mention “experience with at least one OOP-based programming language”. The main thing the companies are looking for in an interview is for a person who can solve problems and the ability to build systems that are highly scalable, performant, and maintainable.

For example, if a website needs an auto-suggestion system for its search bar, then the Software Engineer writes a code that takes text as input and produces an array of suggestions, efficiently. Then the software developer uses the API to send the text and display the suggestions to the user.

So, for building an auto-suggestion API that can handle high Queries per second(QPS), the use of data structures, algorithms, and a plethora of other concepts like Memory Management taught in Operating System class, distributed computing, databases, etc., are a must.

Who is a Programmer or Coder?

A programmer is a person who can write code in general. It doesn’t matter whether you can build Tesla’s autonomous driving system, create websites, or participate in hackathons.

If you can write a code, then congratulations!🥳🎉 You are a Programmer now.

Disclaimer: Sorry to break your heart 💔 but HTML and CSS doesn’t count 😑.

Conclusion

Software Developer: Math and Computer Science topics are not needed but can help in the long run.

Software Engineer: Math and Computer Science topics are must. No excuses! No shortcuts! Although there are multiple frameworks for every small thing these days, still you are expected to learn the internals or at least have a high-level understanding just in case to tear it and modify it to suit your needs.

Programmer: A Superset of Software Developer and Engineer. Nothing more than that. And please don’t call yourself a “Programmer” or “Coder”. It is really cringy to hear. Especially, experienced developers and engineers hate it and look like you are an amateur. So, never use this word in the software industry!

For reference, the below articles are examples of what Software Developers are expected to have knowledge about:

• Cookies Visualized: Everything you ever need to know about Web Cookies
• React Custom Hooks: A simple intuition if you still can’t hit it off
• JavaScript: Writing your own non-blocking asynchronous functions
• WTF is Synchronous and Asynchronous!
• ReactJS Intro: With and without JSX and Babel for beginners
• ReactJS Intro: Using Babel on the fly during production vs compiling during development
• SQL vs NoSQL: A Senior Software Engineer’s perspective
• Git: Commands you need to know when you are messed up😨!

If you look at the titles of articles above, you realize that those are language and platform-specific, just like how I told you in the beginning.

For reference, the below articles are examples of what Software Engineers are expected to have knowledge about:

• HTTP Server: Everything you need to know to Build a simple HTTP server from scratch
• Software Engineering 101: Time zones and Working with Dates
• Software Engineering 101: Stuffing multiple values into a single 64bit value
• SQL vs NoSQL: A Senior Software Engineer’s perspective
• Deep Learning: Deep guide for all your matrix dimensions and calculations!
• Don’t be confused between Concurrency and Parallelism
• OOP: Everything you need to know about Object Oriented Programming
• WTF is Synchronous and Asynchronous!
• STOP IT! There are no Compiled and Interpreted Languages!
• Git: Commands you need to know when you are messed up😨!

So, first, decide what you want to become and then try to concentrate on things which are necessary. If are interested in UI/UX and your ultimate goal is to be a front-end developer then wasting your time learning algorithms and data structures is impractical. And the same goes for the back-end developers. If your work is to query the database, use third-party APIs and scheduling tasks, etc., then learning about memory management, distributed computing is futile.

But learning a new thing is never a waste of time. It will always come in handy when needed

So, go ahead and bookmark it, and don’t be selfish! Share it with fellow developers to help them out

Want to have a talk with me? Twitter.com/@SkrewEverything.

Liked it? 👏👏👏 it and Share it.

Are Math and CS knowledge really necessary for developers, engineers, and programmers? was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

Web Cookies are more or less like a child 👶. Most of the time, they both are harmless.

But once you don’t give enough attention and leave cookies alone, they can bring down even the most secure websites. By the time you realize, your website is compromised and burned to the ground.

Never think web cookies (or kids) are harmless. Give maximum attention for your better future!

The most confusing things about the cookies are—

1. When they are sent
2. When they are not sent
3. To which domains they are sent
4. How cookies work for cross-domain requests

Let me put an end to all these by visualizing how cookies work in the wild!

The basics of a cookie —

When you first visit a website that you have never visited, the browser doesn’t have any cookies associated with that website.

1. So, the initial request doesn’t have any cookies.
2. Then in the response from the server, a cookie is sent to the browser. The browser saves this received cookie for future requests.
3. From now onwards, whenever the client makes a request, the browser attaches the saved cookies and sends it to the server

This is how the basic concept of cookies work.

There are many important things that most of you don’t realize —

FYI, Cookies are a feature of the browser. For server, it is just a normal header.

The browser takes care of storing and attaching cookies based on the domain(we will come to this) and the expiration time.

If you are familiar with cURL, you might remember that handling of cookies(storing from the response and attaching to the request) must be done by the user by using -c flag.

From cURL man pages:

-c, — cookie-jar <filename>
(HTTP) Specify to which file you want curl to write all cookies after a completed operation. Curl writes all cookies from its in-memory cookie stor-
age to the given file at the end of operations. If no cookies are known, no data will be written. The file will be written using the Netscape cookie
file format. If you set the file name to a single dash, “-”, the cookies will be written to stdout.

This command line option will activate the cookie engine that makes curl record and use cookies. Another way to activate it is to use the -b,
 — cookie option.

If the cookie jar can’t be created or written to, the whole curl operation won’t fail or even report an error clearly. Using -v, — verbose will get
a warning displayed, but that is the only visible feedback you get about this possibly lethal situation.

If this option is used several times, the last specified file name will be used.

Also, if you observe the request and response headers for a cookie, you will see that not all the information is sent to the server. Only the name and value of the cookies are sent.

This is because, the browser is responsible for managing cookies, and the cookie’s attributes like domain, path, secure, httponly, and expiration time help the browser manage its store of cookies. Therefore, although you can read the name and value of a cookie, you cannot read the cookie’s other attributes(through javascript on the client-side). And also when the browser sends the cookie information to the server, the browser only includes the name and value.

Cookies are always sent to the domain(or sub-domains) that it was created by!

This is the most important rule the cookies follow —

No matter what, the cookies are only sent to the domain it was set by, even during cross-origin requests, PERIOD.

If your browser is making two requests for two different domains, then —

Cookies are sent to their respective domains only.

The below scenario is not possible.

Again, to be clear, cookies set by A-DOMAIN.COM will never be sent to B-DOMAIN.COM

This is a major security feature that prevents cookies to be read by other domains. This is necessary because cookies are used for Sessions(to store the sessionID of the user). If this is allowed then it leads to Session Hijacking.

For those of you who don’t know what a Session Hijacking means, the attacker steals the session cookie to pose as the victim without needing to authenticate(without entering/knowing the victim’s password).

Session Hijacking is just one possible threat. Not the only one though.

The server can only set cookies for its own domain —

The thing with the cookies is, the browser only accepts the cookies with the domain name attribute with the same value as the domain name it was sent from.

That means facebook.com can only set cookies for facebook.com.

If google.com tries to set cookies for facebook.com by providing Domain=facebook.com in the Set-Cookie header, then the cookie is rejected by the browser —

Till here, I hope the concepts are clear. If not, make sure you understand them clearly because the whole thing gets complicated as soon as sub-domains are introduced.

Domain Matching

The most confusing thing about cookies is how to share and not to share cookies between your sub-domains

Before getting down the rabbit hole, memorize the below anatomy.

We also need to know how domain matching works in the browsers.

Domain matching is used to decide whether the browser should accept or reject a cookie with a particular domain attribute in response and which cookies need to be sent while performing a request.

The domain matching test always tries to match the suffix of the host name with the domain attribute. If the domain attribute is present in the host name as the suffix, then the domain matching test is passed and cookies are sent(in request) and cookies are accepted(in response).

Another important detail about cookies that most people don’t know is —

Cookies are not affected by Port numbers.

Cookies can work even for IP Addresses as host name.

Below example table shows which domains are accepted and rejected for domain attribute for example.com host in a browser response —

Below example table shows to which hostnames which cookies are sent with a particular domain attribute for a request —

First, let's see how to share cookies —

To share cookies between sub-domains and second-level domains, domain attribute must be set.

Parent domain can set cookies which can be used by sub-domains

If the parent domain sets cookies using a domain attribute then the cookies can be accessed by all of its sub-domains.

Sub-domains can set cookies not to share it with the parent domains but can be shared with its relative subdomains

If a sub-domain(x.example.com) sets a cookie using the domainattribute with the sub-domains hostname(x.example.com), then the parent domain(example.com) cannot access it. But, the sub-domains(y.x.example.com) of it can access it as it becomes parent(x.example.com) to other relative subdomains(like y.x.example.com, z.y.x.example.com, zy.x.example.com etc).

Now, How not to share cookies using HostOnly Cookies

Sharing cookies is not always desired. Sometimes, a particular cookie must be made available exclusively to the host to which it set the cookie.

This can be achieved by not mention anything in the domain attribute in the Set-Cookie header. This makes the cookie hostOnly , which means the cookie is only available to the host it created.

How Secure Flag works in Cookies

Secure flag in Cookies can be used to only transfer the cookies if and only if the connection is Secure ie., over HTTPS. If the connection is not secure, then no cookies are transferred between the client(browser) and the server.

This can help to prevent Man In Middle Attacks in which, a person reads/modifies the unsecured content while transferring between two people without their knowledge.

From Wikipedia:

Secure cookie

A secure cookie can only be transmitted over an encrypted connection (i.e. HTTPS). They cannot be transmitted over unencrypted connections (i.e. HTTP). This makes the cookie less likely to be exposed to cookie theft via eavesdropping. A cookie is made secure by adding the Secure flag to the cookie.

If the connection is not Secure i.e., over HTTP, then the cookie is not accepted(or not sent to the server.)

If the connection is Secure i.e., over HTTPS, then the cookie is accepted(or sent to the server.)

How HTTPOnly Flag works in Cookies

If the httpOnly flag is enabled on a cookie, then it is not accessible to client Javascript.

This is useful when you don’t want anyone to read/modify the cookie on the client-side. For example, a user session cookie doesn’t need to be available for reading and modifying on the client-side.

From Wikipedia:

Http-only cookie

An http-only cookie cannot be accessed by client-side APIs, such as JavaScript. This restriction eliminates the threat of cookie theft via cross-site scripting (XSS). However, the cookie remains vulnerable to cross-site tracing (XST) and cross-site request forgery (CSRF) attacks. A cookie is given this characteristic by adding the HttpOnly flag to the cookie.

How Session Cookies work

Every cookie can be provided with an expiration DateTime either using maxAge or expires attribute. After the expiration DateTime, the cookie gets invalidated/removed from the browser.

But if no value is provided for maxAge or expires attribute, then the cookie is said to be a session cookie.

From Wikipedia:

Session cookie

A session cookie, also known as an in-memory cookie, transient cookie or non-persistent cookie, exists only in temporary memory while the user navigates the website. Web browsers normally delete session cookies when the user closes the browser. Unlike other cookies, session cookies do not have an expiration date assigned to them, which is how the browser knows to treat them as session cookies.

As there is no expiration datetime is mentioned for a session cookie while creating it, the browser deletes this cookie when the browser window(not a tab) is closed/quit.

Cookie behavior in Ajax Requests

While making ajax requests, we can choose whether to send cookies are not using credentials property.

credentials has 3 values:

1. omit
2. include
3. same-origin

You can send requests to either to the same origin or cross-origin using ajax.

Same-origin requests

Same-origin requests are when you make an ajax request to the same hostname displayed in the address bar of a browser.

credentials: 'omit'

While making the request, the browser doesn’t send any cookies to the server.

This option means “don’t send any cookies in the request”.

credentials: 'same-origin'

While making the request, the browser sends available cookies to the server.

This option means that “send the cookies if and only if the request is to the same origin”.

credentials: 'include'

While making the request, the browser sends available cookies to the server.

This option means “send the cookies with the request”.

Cross-origin requests

Cross-origin requests are when you make an ajax request to a server that has a different hostname from what is displayed in the address bar of a browser.

I have seen many people thinking that in cross-origin requests, cookies from different origins are also sent, which is completely wrong!

The above rules still hold, even for cross-origin requests! Again, this is never possible —

All the rules about domain matching still hold in cross-origin requests.

credentials: 'omit'

While making the request, the browser doesn’t send any cookies to the server.

This option means “don’t send any cookies in the request”.

credentials: 'same-origin'

While making the request, the browser doesn’t send any cookies to the server.

This option means that “send the cookies if and only if the request is to the same origin”.

credentials: 'include'

While making the request, the browser sends available cookies to the server.

This option means “send the cookies with the request”.

To make the above thing easy, remember this matrix:

So, go ahead and bookmark it, and don’t be selfish! Share it with fellow developers to help them out

Want to contact me? Twitter.com/@SkrewEverything.

Found any mistakes or got anything to say? Comment down below.

Liked it? 👏👏👏 it and Share it.

References

• How to get the domain value for a cookie in Javascript?
• How to get cookie's path using javascript
• Can I see cookie domain specificity in JS?
• How do browser cookie domains work?
• ASP.NET Cookies Overview
• Can subdomain.example.com set a cookie that can be read by example.com?
• What's in a Domain Name: Sub, Second-Level, Top-Level, & Country Code Domains What's in a Domain Name? All about TLDs!
• RFC 6265 - HTTP State Management Mechanism
• HTTP State Management Mechanism

Cookies Visualized: Everything you ever need to know about Web Cookies was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

Generally speaking, cloud helps us to optimize computing costs based on our demands and takes away unnecessary overhead of purchasing, maintaining and forecasting hardware needs. But cloud makes it difficult for us to test our apps locally. Good thing is, Google provides a bunch of emulators to help us test our apps locally without incurring cloud costs.

But if you are pulling an all-nighter and not careful, your GCP bill will be on 🔥🔥🔥

So, to avoid a heart attack at the end of the month, you need to devote your maximum attention while setting up the emulators. BTW, setting up the emulators is not hard but doing it right is crucial.

Emulators provided by the Google Cloud Platform are —

1. Firebase Functions
2. Firebase Hosting
3. Firebase Firestore
4. Firebase Realtime Database
5. Google Cloud Spanner
6. Google Cloud BigTable
7. Google Cloud PubSub
8. Google Cloud DataStore

Setting up the above emulators require zero to minimum code change depending on from where(client or server) your code is using GCP.

In this part, we look at setting up Google Cloud Platform Emulators. For Firebase Emulators, check out this part —

[Firebase] Everything you need to know to setup Firebase and GCP emulators — Part 1

Google Cloud Platform

To see the list of emulators and other components provided by GCP, use this command —

gcloud components list

Note: Firebase and GCP offers same emulators for pubsub and firestore. You can install it using either one of them.

Note: Even though GCP provides an emulator for Spanner but it is a docker image. Spanner emulator native binary is only available for Linux. The above image is from macOS. That is the reason there is no Spanner emulator in the list. When you start the Spanner emulator, gcloud automatically downloads docker image if there is none present.

From Google Spanner emulator documentation:

Use the cloud spanner emulator docker image even if the platform has a native binary available in the sdk. Currently we only provide a native binary for Linux. For other systems, you must install docker for your platform before starting the emulator.

To install all the emulators —

gcloud components install pubsub-emulator cloud-datastore-emulator cloud-firestore-emulator bigtable

Managing SDK Components | Cloud SDK Documentation | Google Cloud

To start the GCP emulators —

gcloud beta emulators [emulator_name] start [flags]

Unlike Firebase, GCP doesn’t provide a nice json file to configure ports. To give your desired port, you need to pass it as command line argument while starting the emulator. If you are feeling lazy, gcloud gives default port values to each emulator.

Pubsub:

Note: If you have started pubsub using firebase tools, then no need to do it using gcloud

gcloud beta emulators pubsub start

• Testing apps locally with the emulator | Pub/Sub | Google Cloud
• gcloud beta emulators pubsub start | Cloud SDK Documentation

Firestore:

Note: If you have started firestore using firebase tools, then no need to do it using gcloud

gcloud beta emulators firestore start

gcloud beta emulators firestore start | Cloud SDK Documentation

Bigtable:

gcloud beta emulators bigtable start

• Using the Cloud Bigtable Emulator | Cloud Bigtable Documentation
• gcloud beta emulators bigtable start | Cloud SDK Documentation

Spanner:

gcloud beta emulators spanner start

If you are using other than Linux, then you have to install and start Docker before starting the Spanner Emulator.

If you don’t have Spanner Docker image, then a new one is downloaded and started.

If you already have it, then you will see something like this —

• Emulate Spanner locally | Google Cloud
• gcloud beta emulators spanner start | Cloud SDK Documentation

DataStore:

gcloud beta emulators datastore start

Sometimes you might encounter this error —

Don’t worry though. It requires a Project to run the datastore. You have 2 options here.

1. Export a random string to export CLOUDSDK_CORE_PROJECT='my-project-id' . This is not advisable because as this may make datastore not work with other emulators without additional setup.
2. You need to login and choose your project. Don’t worry, this will not make your code to hit production servers. You can do it with confidence.

After doing one of the above —

• Running the Datastore Emulator | Cloud Datastore Documentation
• gcloud beta emulators datastore start | Cloud SDK Documentation

Now that we have installed all the required emulators in our working machine and everything is running, next we see how to use those emulators instead of our production GCP servers.

Using Emulators

The above visualisation should be self explanatory on how the emulator works.

GCP

Pubsub, Spanner, Datastore and Bigtable can only be accessed from your GCP or other servers but not from client side.

Unlike Firebase, you don’t have a json file pointing to the local emulators. Just like how we gave the port number through command line while starting the emulators, we need to export that same port numbers to point our services to our emulators.

Pubsub:

Executing this command shows the environment variables you have to export in order to point your services to local emulators.

gcloud beta emulators pubsub env-init

Execute the command returned by the command.

From Google documentation:

On the machine that runs your application, set the PUBSUB_EMULATOR_HOST environment variable and value as directed by the output of the env-init command. You can optionally set the PUBSUB_PROJECT_ID environment variable for the project you want to use for the emulator, but you only need to set PUBSUB_EMULATOR_HOST for your application to connect to the emulator.

gcloud beta emulators pubsub env-init | Cloud SDK Documentation

Bigtable:

gcloud beta emulators bigtable env-init

gcloud beta emulators bigtable env-init | Cloud SDK Documentation

Datastore:

gcloud beta emulators datastore env-init

• gcloud beta emulators datastore env-init | Cloud SDK Documentation
• gcloud beta emulators datastore env-unset | Cloud SDK Documentation

Spanner:

gcloud beta emulators spanner env-init

gcloud beta emulators spanner env-init | Cloud SDK Documentation

Firestore:

export FIRESTORE_EMULATOR_HOST=localhost:$PORT

That’s it

You must be feeling tired going through all those commands, codes and links.

So, go ahead and bookmark it and don’t be selfish! Share it with fellow developers to help them out

Want to contact me? Twitter.com/@SkrewEverything.

Found any mistakes or got anything to say? Comment down below.

Liked it? 👏👏👏 it ( come on! If I can spend this much time to write then you can spend few seconds to clap it 50 times 🤷) and Share it.

[GCP] Everything you need to know to setup Firebase and GCP emulators — Part 2 was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

Generally speaking, cloud helps us to optimize computing costs based on our demands and takes away unnecessary overhead of purchasing, maintaining and forecasting hardware needs. But cloud makes it difficult for us to test our apps locally. Good thing is, Google provides a bunch of emulators to help us test our apps locally without incurring cloud costs.

But if you are pulling an all-nighter and not careful, your GCP bill will be on 🔥🔥🔥

So, to avoid a heart attack at the end of the month, you need to devote your maximum attention while setting up the emulators. BTW, setting up the emulators is not hard but doing it right is crucial.

Emulators provided by the Google Cloud Platform are —

1. Firebase Functions
2. Firebase Hosting
3. Firebase Firestore
4. Firebase Realtime Database
5. Google Cloud Spanner
6. Google Cloud BigTable
7. Google Cloud PubSub
8. Google Cloud DataStore

Setting up the above emulators require zero to minimum code change depending on from where(client or server) your code is using GCP.

In this part, we look at setting up Firebase Emulators. For Google Cloud Platform emulators, check out this part —

[GCP] Everything you need to know to setup Firebase and GCP emulators — Part 2

Installing Emulators —

Firebase

If you haven’t setup your firebase project already in your working directory, go ahead and do it with the command —

firebase init

You can install the emulators while initialising your firebase project or later with the command —

firebase init emulators

Firebase CLI provides 5 emulators to install. You can install whatever emulators you like from the list. After installing the emulators, in your firebase project directory, you can see firebase.json file. That contains all the info about your emulators. You can change the port numbers and path to your rules json files.

Note: Firebase and GCP offers same emulators for pubsub and firestore. You can install it using either one of them.

Note: I have intentionally skipped installing Hosting emulator. You’ll know why in a moment.

To run the firebase emulators —

# To start all emulators
firebase emulators:start

# To start only selected emulators
firebase emulators:start --only functions firestore [other emulators]

Let’s go through each one of them —

1. It shows the list of emulators started
2. Do you still remember that I didn’t setup a Hosting emulator? This is what I want to show you. Firebase CLI is kind enough to warn that “Hey! for whatever reason, you are not running [this] emulator. So, be careful as your test runs will hit your production servers”
3. Those are the logging files for each emulator
4. These URLs are the cloud functions URLs for your local emulator. These can be used to call your cloud functions from client side code.
5. This one uses “triggers”. I have setup a function trigger to be run after every upload to Cloud Storage. The thing is, firebase doesn’t provide a Cloud Storage emulator, so the cloud function is being ignored. If you have a Cloud Firestore triggers, it will be executed only if you enable the Cloud Firestore emulator.
6. You can visit the mentioned URL to visit the UI of Firebase Emulators.

If you look at the image carefully, you can see Firestore and Realtime Database tabs. You can use those tabs to insert or view your data just like you do it in the Firebase Console.

But there is a bug in Firestore UI where the data is being written from the Firebase SDKs but you can’t see the data visually. I have commented a workaround on this bug. Check out if you experience the same bug —

Firebase real time database at local emulator is not showing updated data · Issue #2322 · firebase/firebase-tools

For example if you have created a document at the path in your code —

list/users/1003/details

Then create a dummy document in the same path through Emulator UI —

list/users/dummy

Now you can see your original data.

Install, configure and integrate Local Emulator Suite | Firebase Local Emulator Suite

Now we are done with Installing all Firebase specific emulators. Let’s move on to using the emulators.

Using Emulators

The above visualisation should be self explanatory on how the emulator works.

Firebase

There are 2 cases you have to consider while using firebase emulators in your code —

1. Client-side code accessing Firebase products
2. Server-side code accessing Firebase products

Firebase Client-side code —

If you are using Firebase products, you can access the backend directly from the client code using Firebase Client SDKs.

There is one important step to follow to make sure the Firebase Client SDKs are hitting emulators — Point the Firebase Client SDK to your localhost:port.

Firebase Functions:

In Firebase Functions, there are 3 types to invoke a function:

1. Callable Functions
2. HTTPS Functions
3. Trigger Functions

Callable Functions:

While initialising the functions, pass the localhost:port

Link:

• Call functions from your app | Firebase
• Connect your app to the Cloud Functions Emulator | Firebase

HTTPS Functions:

If you are using axios, fetch etc type of libraries to run the functions, use the URL returned by the Firebase CLI

Use the URL returned by Firebase CLI tool like mentioned in 4.

Trigger Functions:

There is no setup required for using trigger functions.

Firebase Firestore:

While initialising the Firestore, pass an object to point it to the localhost —

Link:

Connect your app to the Cloud Firestore Emulator | Firebase

Firebase Realtime Database:

While initialising the Firebase object, pass an object to point it to the localhost —

Link:

Connect your app to the Realtime Database Emulator | Firebase

Firebase Server-side code —

If you are using Firebase admin SDK to interact with the firebase products then the good news is, you don’t have to do any additional work to make it work with the Firebase Emulators.

Remember the firebase.json? It is used by the Firebase admin SDK to find out in which port our emulators are running. If any emulator is not running, then Firebase admin SDK hits your production server (remember the warning about Hosting emulator not running while starting the emulator?). If the emulator is running, then the SDK hits your emulator. How convenient ❤️

That’s it

You must be feeling tired going through all those commands, codes and links.

So, go ahead and bookmark it and don’t be selfish! Share it with fellow developers to help them out

Want to contact me? Twitter.com/@SkrewEverything.

Found any mistakes or got anything to say? Comment down below.

Liked it? 👏👏👏 it ( come on! If I can spend this much time to write then you can spend few seconds to clap it 50 times 🤷) and Share it.

[Firebase] Everything you need to know to setup Firebase and GCP emulators — Part 1 was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are some things which every software engineer should know but unfortunately it is not taught in the schools. One of them is: Time Zones and Working with Dates.

Software Engineering 101 Series —

1. Stuffing multiple values into a single 64bit value
2. Time Zones and Working with Dates(you are here)

The most important thing in any app with a database is about getting the dates and timestamps right. You might not experience any problems when your app only caters single timezone, but if your app is used by people all around the world, you need to be very careful about getting the dates and timestamps right for your users who are in different time zones around the world.

There are 2 different types to mark an instant of time —

1. Timestamp
2. Datetime

Timestamp —

Epoch, also known as Unix timestamps, is the number of seconds (not milliseconds!) that have elapsed since January 1, 1970 at 00:00:00 GMT (1970–01–01 00:00:00 GMT).

You might be thinking why January 1, 1970 at 00:00:00 GMT is the starting point in computer systems. Well, I could explain why but it completely deviates from the context of our article. So, below links have all the details about “why”.

• Why are dates calculated from January 1st, 1970?
• Why is 1/1/1970 the "epoch time"?

Another important thing is — Unix Epoch Common, But Not Dominant. There are many epochs which are used by various fields. If you are interested to know about them, follow the below link —

Why are dates calculated from January 1st, 1970?

But the most crucial thing to remember is, it is gonna work only till January 09, 2038 03:14:07 GMT (2038–01–09 03:14:07 GMT). Why? Because in most of the systems Unix timestamps use signed 32-bit wide integer to represent the seconds. This is known as Year 2038 problem

Although default unix timestamps are signed 32-bit integers, programming languages like Java can return a timestamp with a 64-bit integers. But most of the softwares like MySQL still uses the signed 32-bit integers to represent timestamps and it will return 0 after 03:14:07 UTC on 19 January 2038.

More about this MySQL behaviour below —

MySQL Bugs: #12654: 64-bit unix timestamp is not supported in MySQL functions

But it won’t be a problem if you are using a 64-bit systems, because by the time 2038 year arrives, it will be shifted to signed 64-bit integer by major software providers. The problem will be with the embedded systems which use 32-bit systems like military, aviation etc. More about that below —

Year 2038 problem

The problem you should be anticipating as a Software Engineer is when you want to work with or store a time after January 09, 2038 or before January 01, 1970.

This problem has been experienced by AOLServer in 2006:

In May 2006, reports surfaced of an early manifestation of the Y2038 problem in the AOLserver software. The software was designed with a kludge to handle a database request that should “never” time out. Rather than specifically handling this special case, the initial design simply specified an arbitrary time-out date in the future. The default configuration for the server specified that the request should time out after one billion seconds. One billion seconds (approximately 32 years) after 01:27:28 UTC on 13 May 2006 is beyond the 2038 cutoff date. Thus, after this time, the time-out calculation overflowed and returned a date that was actually in the past, causing the software to crash. When the problem was discovered, AOLServer operators had to edit the configuration file and set the time-out to a lower value.

Players of games or apps which are programmed to impose waiting periods are running into this problem when they attempt to work around the waiting period on devices which harbor the coding, by manually setting their devices to a date past 19 January 2038, but are unable to do so, since a 32-bit Unix time format is being used.

Remember: Only use the timestamps if and only if you want to store the current date and time. Don’t use it to work on or to store future or past explicitly

DateTime —

Datetime is used to store the exact date and/or time, like a string. Unlike timestamp, which always stores in UTC, datetime has its own time zone. So, you need to be careful while storing a datetime.

Datetime can be in the form “YYYY-MM-DD” or “YYYY-MM-DD HH:MM:SS”. Though the name suggests it stores date and time together, you can store only the date also by making “HH:MM:SS” all zeros like — “YYYY-MM-DD 00:00:00”

As I already told you, datetime values are stored just like a string literals without any information about the time zone it belongs. For example, if you store the datetime in New York and then access it from Tokyo, then the datetime will represent the New York’s time zone, without any explicit info that it is a New York’s time zone. So, unless the person who is storing the datetime provides the time zone info, nobody can know about the time zone used to store it for sure.

Remember: Always store datetime in UTC! Never store it in your local or any other time zone

Always storing the datetime in UTC makes it very easy to reason and when you want to display it to your users, just convert from UTC to local time zone.

One more advantage of saving in UTC is — there is no confusion about Daylight Savings.

UTC/GMT/Zulu —

If you are not familiar with dealing time zones, you might be wondering what is UTC/GMT.

UTC stands for Universal Time Coordinated, and it’s the primary time standard by which the world regulates clocks and time. UTC use to be called Greenwich Mean Time (GMT), but still is in English speaking countries, or Zulu Time (Z), which is used in plane and ship navigation. UTC is the time at the Prime Meridian (0° Longitude) given in hours and minutes on a 24 hour clock.

To obtain your local time you need to subtract or add a certain number of hours from UTC depending on how many time zones you are away from Greenwich. The map below shows the standard difference from UTC time to local time.

To know more about the differences between UTC and GMT —

Can You Tell Zulu Time Like A Meteorologist?

The main difference you should be knowing as a software engineer between GMT and UTC is that UTC does not observe Daylight Saving Time

Zulu Time —

You might have seen a datetime like 2020–12–31T20:17:46.384Z.

ISO 8601

This format is defined by the sensible practical standard, ISO 8601.

The T separates the date portion from the time-of-day portion. The Z on the end means UTC (that is, an offset-from-UTC of zero hours-minutes-seconds). The Z is pronounced “Zulu”.

What is this date format? 2011-08-12T20:17:46.384Z

Zulu time zone is used by military services which usually means UTC. You can find other list of time zones used by military services around the world below, if you are curious but we only deal with “Zulu”.

List of military time zones

Choosing a Time zone for your servers —

This is a very important topic but most beginners don’t have enough knowledge.

Always! Always keep your server’s time in UTC

No matter in which time zone your server is in, always configure it to UTC. Because every software, program, programming language, logging systems etc like MySQL, MongoDB, Java, node(JavaScript) will return your server’s time when you ask for datetime or timestamp.

If you are having all your servers in a single time zone, then it doesn’t matter because you can infer about the default time zone based on your server’s region. But distributed servers across different regions will produce a lot of problems like out of order logs and data due to timestamps from different time zones and daylight savings.

Summary—

There is a famous rule while developing any program/software —

Never trust any data coming from the user! Not even from your grandma… Always sanitize user inputs!

Remember this even while storing info about date like when was the data created, posted or edited.

1. Set all your servers’ time zone to UTC.
2. Always use the Server’s time to fill up metadata like created, edited, posted etc.
3. Always store the datetime in UTC time zone.
4. While displaying the datetime to the user, convert it to their local time zone at the application layer.
5. Never use timestamps to store past or future.
6. Try to use datetime format instead of timestamps as it makes visual debugging easy. With timestamp, you have to convert from seconds to date but datetime is like a string. So, if you find yourself accessing the raw values for debugging, then try to use datetime.

Want to contact me? Twitter.com/@SkrewEverything.

Found any mistakes or got anything to say? Comment down below.

Liked it? 👏👏👏 it 50 times and Share it.

Software Engineering 101: Time zones and Working with Dates was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are some things which every software engineer should know but unfortunately it is not taught in the schools. One of them is: How to stuff multiple values into a single value.

Software Engineering 101 Series —

1. Stuffing multiple values into a single 64bit value (you are here)
2. Time Zones and Working with Dates

If you don’t know about this, you may be wondering

1. How does that help?
2. Why can’t I use different variables for each value?
3. Why to complicate things unnecessarily? I have 8GiB of RAM, why should I care about saving 8 bytes?

The main aim of stuffing multiple values into a single large value is not to save memory or decrease the number of variables in the program.

The main aim is to encode multiple useful values into a single value to identify it uniquely, most of the time

Instead of going into “how to do it” first, let’s take an unconventional route and first go into a use case to understand why it is needed and where you can use it. This can make you understand the “how to do it” intuitively.

Use Case

Let’s consider a scenario where this technique is used extensively, database sharding at application level.

Database sharding at an application level means the application itself knows where the data is residing among your swarm of database instances. Instead of having some kind of routing mechanism or hash tables to locate the data on the server, the application knows which database has the data it needs. This type of sharding doesn’t depend on the database you are using and doesn’t matter if the database doesn’t provide any sharding features like MySQL.

For example, you are just starting your website and designing a database for it. Normally, if you don’t have any natural key, you would use the auto-increment value as the primary key. And also you use that auto-incremented ID as a part of the url to map to the data(like how Stackoverflow does).

After 1 year, your database server is struggling to catch up with the traffic requirements. To solve that, you have 2 options.

1. Vertical Scaling
2. Horizontal Scaling

Vertical Scaling is costly and finite. So, you want to opt for horizontal scaling but MySQL doesn’t provide horizontal scaling feature. So, you add a new MySQL instance to shard the data. Now, you need to find a new mapping to find the sharded data directly instead of searching all the instances.

This is where we can use the “stuffing multiple values into the single 64bit value” technique.

Before doing that, we need to do some things:

1. Give every database shard a unique ID and call it as dbID
2. Now, you already have auto-incremented postID for every row in a database.
3. Decide how many bits you want to allocate for each ID above. For example, you allocated 10 bits for dbID and 42 bits for postID.

10 bits can store 2¹⁰ = 1024 values, 42 bits can store 2⁴² = 43,98,04,65,11,104

It means you can create 1024 shards and each shard can have 43,98,04,65,11,104 rows.

So, instead of identifying the post using the postID, you also need to identify shard i.e., dbID.

So, instead of encoding your url like “your_domain.com/posts/postID”, we encode it like “your_domain.com/posts/newID”. Now, the newID contains the info about postID and dbID so that your app knows which shard to ask for the data directly without any middleware.

Now, we are going to encode postID and dbID into the newID like:

(I’m using Java, so Java’s Long is 63 bytes. Why is this important? We’ll see about it below in “how to do it”.)

Disclaimer: Don’t be intimidated if you don’t know what it is. We will talk about how to do it after this.

long dbID = 997;
long postID = 8_04_65_11_104L;
long newID = (dbID << 53) | (postID << 11);

System.out.println("newID: " + newID);
System.out.println("dbID: " + ((newID >> 53) & 0x3FF))
System.out.println("postID: " + ((newID >> 11) & 0x3FFFFFFFFFFL));

The output will be:

newID: 8980194136231510016
dbID: 997 
postID: 8046511104

So, now your new url can be “your_domain.com/posts/8980194136231510016” instead of “your_domain.com/posts/8046511104”

This “8980194136231510016” has the information about the shard ID and post ID.

This is called application level sharding because your application knows that postID: 8046511104 lives in database id: 997. There is no need for middleware which needs to check in which database the postID: 8046511104 is.

I’m pretty sure that now you know why it is useful with a real-world use-case. Now, let’s see how it is done

How to do it

First and foremost thing you need to be careful about the size of Integer in your preferred language.

For example,

Javascript has only one type both for floating and integers i.e., Number. Javascript only uses 53 bits to store the integers but not 64 bits. In that 53 bits, 1 bit is used for sign. So, you can only use 52 bits to encode.

Java has “Long” type which can store 64 bit integers. But unlike C and C++, Java doesn’t have “unsigned”. So, you can only use 63 bits because 1 bit is used for sign.

So assume “B” is your number of bits you can use. In the example above, we have used Java, so B = 63.

NOTE: Whatever you wanna encode, your sum of individual values in bits cannot exceed “B” and also only positive numbers are allowed

Let’s assume we want to store the following values in a single value:

1. A with 15 bits — range can be from 0 to 32, 767 (2¹⁵-1)
2. B with 35 bits — range can be from 0 to 34,35,97,38,367 (2³⁵-1)
3. C with 10 bits — range can be from 0 to 1,023 (2¹⁰-1)
4. D with 3 bits — range can be from 0 to 7 (2³-1)
5. E with 1 bit — range can be from 0 and 1 (2¹-1)

The sum of individual values in bits is 15 + 35 + 10 + 3 + 1 = 64 bits.

But our “B” is only 63. So, you have to let go any one of them to bring it below 64. We will let go “E” , which is 1 bit. Now, our sum of bits will be 63 bits, which is equal to “B”.

This is how we want to arrange all our values at the end

There are 2 parts for this.

1. Encoding
2. Decoding

Encoding —

This is how “A” with 15 bits looks in a 63 bit data type:

We need to move all the 15 A ’s to the leftmost empty bit. We need to left shift by 63–15(A) = 48 bits

A << 48

After the left shift, this is how it looks:

This is how “B” with 35 bits looks in a 63 bit data type:

If you look at the previous image, “A” has already occupied 15 bits from the left side. So, we need to shift all the 35 B’s where 15 A’s end.

We need to left shift by 63-15(A)-35 (B)= 13 bits

B << 13

After the left shift, this is how it looks:

This is how “C” with 10 bits looks in a 63 bit data type:

If you look at the previous image, “A” has already occupied 15 bits and “B” has already occupied 35 bits from the left side. So, we need to shift all the 10 C’s where 35 B’s end

We need to left shift by 63–15(A)–35(B)-10(C) = 3 bits

C << 3

After the left shift, this is how it looks:

This is how “D” with 3 bits looks in a 63 bit data type:

If you look at the previous image, “A” has already occupied 15 bits, “B” has already occupied 35 bits and “C” has already occupied 10 bits from the left side. So, we need to shift all the 3 D’s where 10 C’s end

We need to left shift by 63–15(A)–35(B)-10(C)-3(D) = 0 bits

D << 0

After the left shift, this is how it looks:

Now, we have all the “A”, “B”, “C”, “D” bits at their respective positions like below

Now we want to combine these into one variable. The | operator works by looking at each bit, and returning 1 if the bit is 1 in either of the inputs. So:

0011 | 0101 = 0111

If a bit is 0 in one input, then you get the bit from the other input. Looking at (A << 48), (B << 13), (C << 3)and (D << 0) you'll see that, if a bit is 1 for one of these, it's 0 for the others. So:

encodedValue = (A << 48) | (B << 13) | (C << 3) | (D << 0)

Now that we have completed with the Encoding part, let’s get into Decoding part.

Decoding —

In encoding, we have used left shift( << ) and OR( | ) but we use the opposite in decoding. We use right shift ( >> ) and AND ( & ).

Now we want to unpack the bits. Let’s start with the D. We want to get the last 3 bits, and ignore the first 60 bits.

To do this, we use the & operator, which returns 1 only if both of the input bits are 1. So:

0011 & 0101 = 0001

So, if you want to unpack only “n” bits from right side, you need to perform AND with “n” bits in which all the “n” bits should be 1.

To get “C”, we need to right shift till it reaches the rightmost bit. Right shift “C” with the same number you have used to left shift during encoding and perform AND operation with 10 bits(because “C” is 10 bits) set to 1.

Keep performing the same thing for the rest of “B” and “A”.

Right shift “B” by 13 and perform AND operation with 35 bits(because “B” is 35 bits) set to 1.

Right shift “A” by 48 bits and perform AND operation with 15 bits(because “A” is 15 bits) set to 1.

Now, we have decoded the single encoded value into their own individual values

decodedD = (encodedValue >> 0)  & 0x7
decodedC = (encodedValue >> 3)  & 0x3FF
decodedB = (encodedValue >> 13) & 0x7FFFFFFFF
decodedA = (encodedValue >> 48) & 0x7FFF

The magic numbers —

There is a very high probability that you must be wondering what is that Hex value and how to arrive at that particular value.

In many programming languages, you can use different radixs like 2(binary), 6(hexa), 8(octal) and 10(decimal) to represent numbers.

Here, we are using Hex to represent the numbers. Why? Because it is easy to represent multiple 1 in Hex than in any other radix systems.

For example, if you want a number which has 4 bits set to 1, then in binary it will be represented as 0b1111 and in hex it is represented as 0xF. If you don’t know Hexadecimal, then binary might be easy for you to write but you can’t write 48 1s in binary. But in Hex you can write it like 0xFFFFFFFFFFFF. Instead of writing 48 1s in binary system, we write 12 Fs in hexadecimal system.

So, if you don’t know how hexadecimal system works, you need to learn about that. It’s an interesting topic of its own.

Want to contact me? Contact me on Twitter.com/@SkrewEverything.

Found any mistakes? Comment down below.

Liked it? 👏👏👏 it and Share it.

Software Engineering 101: Stuffing multiple values into a single 64bit value was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

Apple constantly claims “macOS is the most secure operating system” and if you are an average user(just like my past self), then chances are you are in delusion just like younger me(well, I’m older and wiser now🧑‍🎓).

Apple repeats that phrase way too often that subconsciously we believe that macOS is indeed most secure operating system in the world. But the truth is: it is not so secure after all. macOS is just difficult to break for an average programmer but not impossible for any programmer who has enough time and intuition of where to look and what to do.

First, let me clear something. The reason why people think that windows is not secure is because it makes into the news most of the time whenever there was a virus attack around the world. But have you once stopped and considered why?

1. Because many people use cracked versions of Microsoft Windows
2. Because many people install cracked versions of software
3. Because there are no restrictions/warnings when untrusted software is being installed

So, how is Apple doing it differently?

1. To begin with, macOS is free for macs. The major thing is, updates are also free since long time.
2. Using cracked version of macOS on non-macs(Hackintosh) is very tricky and buggy. So, most of them avoid it.
3. Unless you are an average user, you can’t even install softwares from unidentified sites and developers. Apple straight out blocks installation.

So, now you can see why Windows is more susceptible to virus attacks.

Another new feature coming in iOS 14 is, Apple notifies users what the app is doing behind the scenes

I wonder how many developers in the world actually use a specific library intended to use the clipboard and a specific function to copy the contents **accidentally**🤔.

And as usually, the Linkedin’s response was “It’s a bug”

So, now you know how Apple is trying to secure its operating system. It is not secure by default. It just lets users decide and take control instead of letting users use it blindly. If you don’t know, in macOS, whenever an app tries to read a file from a directory, it asks for the user permission to allow it or not.

If android and windows used the same *features*, then both can also become a secure one.

So, what exactly breaking a system mean?

Well, to keep it simple, bypassing the system also comes under Breaking the system.

I wrote a Keylogger for macOS, which bypasses notifying users that it is being used in the settings and actually no password is required to run the keylogger.

For some of you who don’t know, macOS doesn’t prevent anyone from creating a keylogger. But to log the key strokes, the new APIs provided by the Apple comes under Accessibility functionality, which requires user password to run and the name of the app we are running in the Settings -> Security & Privacy section.

The below link takes you to some of known open source keyloggers for macOS on GitHub and you can try that and see. Everything requires the user password to actually run the app and it will be shown in the settings app.

Build software better, together

Well, that’s a pretty good security from Apple.

So, how did I break it?

The trick is simple to break/hack anything. Look at the past versions of the system and try to modify it to break the new updated system

If the engineers are really good and actually had time, they would completely remove it from the systems. But most of the engineers don’t have enough time to remove it. So, as they are good, they just fix it on the surface level and forget about it later or if the functionality is tightly coupled with the systems then they remove every known documentation about it to restrict its further usage. And that’s what apple did. That API is one of the essential APIs and Apple cannot remove it completely. Instead they removed most of the documentation and archived it.

Usually, the API docs for old versions are not properly available on the internet. Most of the companies remove it. But once I coincidentally came across a website which conveniently archives every apple developer docs released, which is maintained by Apple itself.

It has a total of 5219 documents, with the oldest one dated 1985–03–01 (That is some serious dedication of Apple towards its developers).

And going through one by one is not gonna happen. So, as a wannabe coder, I did what every wannabe coder does. Yes, I wrote a script to parse the contents of every document and flagged some documents with particular keywords which would likely appear in the developer docs which might help me to write a keylogger. Now, I reduced the number of documents from around 5,000 to about 500. I can reduce further(by removing documents with low frequency of occurrences), but I don’t wanna take any chances. So, instead of reducing the number, I sorted the documents with the frequency of keywords and started going through one by one.

It probably took over a month to shortlist some documents which can help me to write a keylogger. It might sound stupid but once again I went through the new set of shortlisted documents to further shortlist.

Finally, after 3 months of going through those, I was left with 1 document. Uff… that was a hell of a journey to reach 1 document from 5,000 documents.

The document is: Technical Note TN2187

It was published on 2007–12–18 for developers to develop apps for Mac OS X 10.5 (Leopard)

The keywords that picked up this document is “USB” and “KEYBOARD”. Surprisingly, their occurrences in the document is:

• USB: 6 times
• KEYBOARD: 8 times

If I had further reduced the number of documents by removing low frequency occurrences then I wouldn’t have came across this document for sure. I guess it’s time for another tip:

If you want to break a system, there are no shortcuts. You have to go through every single thing you got. You never know what you missed. Or the best thing to remember while coding anything is: Murphy’s Law

Now, back to why you are here. That document is so old that, most of the links mentioned in that no longer exists. Everything is 404.

Those links are very important because it clearly states that:

So, now I don’t have any references to actually have the basic understanding about how to use this doc. To further complicate things, the docs are in Objective — C, which I have never touched.

But the odds are on my side. I got plenty of time to waste and if you ask any programmer/engineer, they will say that anything could be built, given enough time.

Without a further a due, I started coding it in Swift and logging every line of code to understand what’s it doing exactly. It took well over 2 weeks to decode everything and map every key code generated by that interface to the actual keyboard values.

Finally, after 4 months of persistent work, I completed writing the code for my keylogger. It just took 420 lines of unoptimised code(I swear I did not plan to reach that exact number) to break the most secure OS, the macOS.

Link to the keylogger

I open-sourced the keylogger on 2017–02–01 on GitHub. It became so popular that it was on the trending page of GitHub for multiple days world freaking wide.

SkrewEverything/Swift-Keylogger

And don’t think it won’t steal passwords by seeing the below disclaimer in the Readme in my github link:

It can also be bypassed very easily and putting it on the internet won’t be “for educational purpose” anymore, even if I state it explicitly in the documentation. So, I excluded it from the open-source.

What is the current status of the project?

Unfortunately, it doesn’t work anymore as initially intended on macOS Catalina (10.15 ), which was released on 2019–10–07

From 10.15 onwards, it asks for the user’s permission to run and it shows up in the Settings -> Security & Privacy -> Privacy -> Input Monitoring

But it still works perfectly fine till macOS Catalina, without requiring any permissions to run and also without showing up in the Settings app.

But, am I sad that Apple fixed it?

No, in fact I’m happy. It’s probably a good thing that Apple fixed this. It prevents so many clueless people from being spied on or protecting them from losing important stuff from their personal computers by sniffing.

For example, it stops Linkedin like corporations from spying on people and stealing data without their consent and when caught, they say “it’s a bug”

Hate Apple how much you want to hate for their expensive products or imposing “Apple Tax” on it’s customers but, when it comes to security and privacy, particularly in this age, it’s still worth to pay that “Apple Tax” to own Apple products, at least for important stuff.

**Inner me: I really hope Apple comes across this page and sponsors me for this post for talking good and promoting it🤞**

Summary

If you reached till here, you might be thinking is it really breaking the macOS?

To recap what I said in the beginning —

Well, to keep it simple, bypassing the system also comes under Breaking the system.

I wrote a Keylogger for macOS, which bypasses notifying users that it is being used in the settings and actually no password is required to run the keylogger.

For some of you who don’t know, macOS doesn’t prevent anyone from creating a keylogger. But to log the key strokes, the new APIs provided by the Apple comes under Accessibility functionality, which requires user password to run and the name of the app we are running in the Settings -> Security & Privacy section.

So, yeah… I broke it 🤷

Whether it’s Linux, Windows, macOS or any other Operating system, it doesn’t matter. Every OS has their fair share of never ending bugs. That’s why companies constantly release Security Updates.

The only thing Apple is doing differently is that they are providing maximum details about what the apps are doing to its customers. So, even if the OS fails to protect, users can at least see what’s happening and mitigate the damage ASAP. For example, people wouldn’t know how some companies are stealing data from the clipboard of the users if its not for iOS 14. Or my keylogger stealing from your mac.

Liked it? Then 👏🏻👏🏻👏🏻 it and share it!

My experience of breaking the claim of most secure OS, Apple macOS was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let’s be honest, React Hooks are awesome 😍 but confusing as hell 😵

Previously, I’ve strictly used Class Components instead of Functional Components even if I didn’t need the lifecycles and state functionality as I’m lazy to refactor my code from functional component to class component if by any chance I ever wanted to introduce lifecycles or/and state functionalities in a component.

When React introduced Hooks, I was reluctant to use them as they were confusing, at least when I read the docs initially.

But hey! Time makes everything better and Hooks are no exception

What changed my mind? Well, boredom of this lockdown🥱. I was so bored that I gave another chance to Hooks and after creating a simple app while learning it, it kind off hit me in the head and now I can see everything around me in Hooks🧑🏻‍🎓.

Ok, I was exaggerating for the effect.

So, let’s look at some points, tips, explanations to understand Hooks as simple as possible.

What are React Hooks anyway?

It’s a new feature(well, not so new. It was released in 2018) to use lifecycles and states in Functional components.

Before the introduction of the Hooks, the only way to use lifecycles and states is through Class component.

Why is everyone losing their minds over Hooks anyway?

If you haven’t, then you have to look at these tweets from 2018 first.

Sunil Pai on Twitter

ok so - I took dan's classes/hooks code from react conf, blacked out the 'unnecessary' bits, then colour coded bits by 'concern'. so much nicer. the effect is amplified in more complex components, where concerns are split and mixed across lifecycle methods.

Got it why everyone is losing their minds over Hooks?

If you didn’t then let me tell you what’s going on in the above GIF.

The first part of the GIF uses Class Component and second part uses Functional Component with Hooks. And color codes are used to group lines based on their logic required to do THEIR THING.

If you observe, when using Class Component, the logic of the code is spread all over the component. But, when using Functional Component with Hooks, the logic of the code is confined to their own places.

This greatly increases developer productivity while reading through the code or debugging it. With Hooks, you don’t have to scroll your IDE to relevant sections of the code responsible for its job. If you want to know how a feature works, you just have to look at one place to make sense.

Also, the amount of code to write a component decreases significantly with Hooks as you don’t have to deal with “bind”ings, “constructors”, repeating code in different methods etc.

The best thing about Hooks is: You can create and re-use your custom Hooks in other components, which reduces code duplication substantially.

And also, one more trivial advantage is: you don’t have to shuffle between Class and Functional Components based on your needs. You can just learn Functional Components with Hooks and you can forget about Class Components all together. Unless, you have to maintain your old React code which was written in Classes. But, if you are starting a new project, then you can just write all your components using Functional Components and later if you need lifecycles and state, then use Hooks to introduce them into your already written Functional Component instead of refactoring it into Class Component.

NOTE: If you are a beginner or starting to learn React, then start with the Class Components. Class Components are more detailed and explicit, which helps you to understand the concepts and how things work more concretely. After you grasp how React works, then you can shift to the Hooks.

Before getting into the explanations and intuitions, I hope you know the basic syntax of the useState and useEffect .

If you don’t, then come back after learning the basic syntax. I can explain it though, but there are already numerous wonderful tutorials and articles on the basics of Hooks. Writing them provides no value and diverts the aim of this article, which is to get the intuition but not an another tutorial on Hooks.

The Facebook’s documentation is pretty good actually with some use-cases.

Introducing Hooks - React

Just don’t go deep. Get the basics on how-to use them. Once you get the intuition here, you will get when-to and then you can learn the more advanced concepts like useReduce .

Let me know in the comments if you would like a tutorial on Hooks too.

Getting the intuition

Let’s look at some code first:

If you are familiar with the Class Components, this is how you must be doing your input handling in your components.

So, what’s wrong in that?

First of all, nothing is wrong with that. If it is wrong, then Facebook wouldn’t have introduced it in the first place.

But when you look at the above code after the introduction of the Hooks, you can realise how the logic is spread across the whole component and there’s a lot of code duplication(we will get to this below) in multiple Class Components.

This is how it looks in Hooks

If you notice, the Hooks gist looks more elegant, concise and cleaner than the Class Component gist.

Whatever I mentioned above is aesthetics. Now, let’s get to the most interesting topic: Reducing code duplication through custom Hooks.

Building your own Hooks lets you extract component logic into reusable functions.

Consider a scenario: You have multiple components in your UI, which takes inputs from the users. Before Hooks, you can only use Class Component to handle input data. So, you have to write the samehandleInputs(event) method logic in every component that takes input from the User. If you have 10 different components in your app, then you have to write the same handleInputs(event) method logic 10 times in those 10 components.

But with custom Hooks, you can write it only 1 time and use it in “N” number of components. This reduces code duplication substantially as mentioned before.

I’ve changed the above Hooks code a little to make it a custom Hook.

Can you comprehend what I did there?

I just removed the logic from the component to its own function so that it can be re-used in our other functional components.

Now, you can move the function useHandleInputs() {...} into another file, and can be import ed into your other components.

And now, we have successfully written our first custom Hook.

How can a normal JavaScript function can maintain state of its own?

If you know React, then this will be your first question after seeing the code above.

Just remember one thing about React Hooks,

React Hooks are plain JavaScript functions, which can maintain their own state and lifecycles, isolated from components.

That’s it! That’s what you need to know to get the intuition on how-to, when-to and what-to use custom Hooks for.

Those custom Hooks works exactly like when a Class Components update it’s state.

The main thing you have to do is, return the state s from your custom Hook to the functional component you want to use it and that’s it.

Whenever the state inside your custom Hooks is changed, your component is re-rendered to reflect the new changes.

If you still haven’t got it, then try to reason this simple code that we always do and also a common practice in React.

We send the states and callbacks from the Class Component to Functional Component as props and whenever a change happens in the Class Component states, the new states are passed to Functional Component as props to re-render to reflect the new state.

If we want to compare our custom Hooks with the above code, then assume:

The code sections are color coded to group how similar they are in functionality.

Yellow: Maintains the state and passes it to Purple components

Purple: Purple doesn’t have it’s own state but depends on the Yellow’s state to render

In Class Component, the state and lifecycles are tied to its component. So, it can be re-used as a component but can’t be re-used as a logic.

In Custom Hooks, the state and lifecycles are not tied to any component. Custom Hooks can exist individually — logically as a function. So, it can be re-used as a logic in other components.

A tip for writing your own Hooks easily

I know custom Hooks are cool and awesome but don’t always try to write a custom Hook.

First, use the Hooks provided by the React in a Functional Component. Then if you find that it can be re-used in another component, then try to extract it outside of that Functional Component like we did in the above:

1. We wrote a Functional Component App using useState Hook.
2. We found out that we can use the handleInputs() function in other Functional Components also.
3. Extracted that function from our App Functional Component to its own function useHandleInputs() , which is our custom Hook.
4. Import our custom Hook useHandleInputs() into App Functional Component or any other Functional Components.

Also, you might find in most of the articles/tutorials that custom Hooks are written using useState()and useEffect() in combination always. Writing custom Hooks doesn’t require you to use both useEffect() and useState() Hooks compulsorily. You can use any combination of Hooks. You can either use useEffect() or useState() . You can use both of them. You can even use your own custom Hook to write another custom Hook.

More Resources

React docs has explained custom Hooks with state and lifecycle with a use-case, which is pretty simple and useful. Check it out 👇🏻👇🏻👇🏻

Building Your Own Hooks - React

And don’t forget to checkout the 2 rules you have to follow while using Hooks. React docs has pretty good explanation for this also 👇🏻👇🏻👇🏻

Rules of Hooks - React

I hope this helped you to get the intuition on “How to write your own Hooks”

If you still got any doubts, then comment down🙂

Liked it? Then comment down, 👏🏻👏🏻👏🏻 it and share it!

React Custom Hooks: A simple intuition if you still can’t hit it off was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are 1000s of articles on “The difference between SQL vs NoSQL” and we don’t need more of those articles but still many people keep writing the same points in every articles. As a result, many beginners can recite the pros and cons like a parrot but tend to struggle while deciding when to use and which one to use.

The problem is not with the beginners but with those redundant articles which contribute no value.

The struggle to decide is inevitable even if you read 100s of articles more because everything is just a copy and paste of previous articles.

I’m not gonna repeat all those redundant points here. Instead, let’s look at the “SQL vs NoSQL” in a different perspective — A Senior Software Engineer’s perspective.

In this article, I’m gonna talk specifically about MySQL vs. MongoDB as they are the most popular ones.

So, When you should use MySQL?

MySQL is not just a database but a relational database. If you want to enforce some relations between the data, then you need a relational database i.e., MySQL.

Seems little vague right? Lets see what you can actually achieve from this relational databases that everyone keeps talking.

Consider a scenario where you have built a webapp where a User can comment on Posts.

This is the minimum schema for that table:

So, what type of constraints we can think of?🤔

1. userName should be unique for every user.
2. Only users present in the Users Table can write a post
3. Only users present in the Users Table can comment under the posts present in the Posts Table.

Can you guess a general flow for every table for inserting new rows?

For Users Table:

For Posts Table:

For Comments Table:

So, this is how the naïve approach is when there are no relationships in the MySQL.

We need to check all the constraints in our application server. And this is what you need to do in MongoDB and in MySQL(if you don’t create any constraints or relationships).

This is too much code to write, makes too many requests to database and the transfer of data between database server and application server. Also, there is a possibility of introducing bugs in the code and becomes hard to maintain the code if the complexity of the constraints and relationship increases.

Let’s see how to simplify this process and delegate this whole checking to the database server.

Just introduce some relationships into the schema and you are good to go

The above relationships states that:

1. userName is unique(defined in the schema but you can’t see here visually)
2. A user in the Users Table can post any number of Posts
3. A user in the Users Table can Comment any number of times on any number of posts present in the Posts Table.

So, if a user is not present in the Users Table tries to post a Post, then MySQL automatically throws an Error and no insertion takes place and same thing happens with the Comments Table if either user or post is not found in their respective tables.

Now, instead of checking for whether an insertion can be done explicitly in the application server by you, MySQL takes care of that for you. So, you can just execute only one INSERT command instead of executing 2–3 commands for insertion operation.

Here is the schema used to define tables and relationships and constraints:

Consider another scenario where if the user deletes their account, we have to delete all their posts and comments from all the tables.

In MongoDB, you have to write a command to delete all the posts and comments with the userID, who deleted their account. If you make a mistake, there is a probability to delete the posts and comments with userID as NOT it’s owner. You might think “this is a simple operation and how can I screw up?”. This is just a simple example. In large production systems, you have 1000s of tables and you have to explicitly mention every one of them to delete it correctly. If you miss a single table/collection, then you start to have orphan records, which leads to data integrity problems.

In MySQL, while creating foreign keys, you can provide extra information on what to do if the parent record is edited/deleted. It is called “CASCADING”

You can read more about here👇🏻👇🏻👇🏻

Difference between On Delete Cascade & On Update Cascade in mysql

Now, you can delegate this responsibility to the server, like what to do when you update and what to do when you delete the parent record. This helps to reduce the bugs or more accurately prevents us from introducing unnecessary bugs. This will in turn helps us to maintain data integrity.

A new thing to remember for you,

MySQL Database actually holds business logic in them

If anyone says “Databases are not supposed to have business logic”, they are wrong or they don’t understand the concept of relational databases. So, next time, hit them with this explanation.

So, When you should use MongoDB?

There are use cases for MongoDB where it excels and where we don’t need the overhead(we will talk about this below) of a SQL.

MongoDB can be used for storing Sessions data/notification systems etc…

In MongoDB, there are no relationships, no normalization, no schema.

So, to store the data of a Session, which doesn’t require any relationships, doesn’t require normalization and schema can change based on what type of data you want to store and there is no need to parse the retrieved data(in JavaScript) as the data returned from the MongoDB is JSON. The same concept applies for creating a notification system where a document is a userID and fields are numbered sequentially for each notification or can be saved in an array. You can just query the MongoDB for new notifications using userID and display them directly, without any processing.

Consider a scenario where you are storing the Session data in the database and like every other session, you need to delete those records after some time for the expiration of the Session.

Can you guess a general flow for deleting the existing expired records?

You need to write a background program or write a function which needs to run for every “n” seconds.

But in MongoDB, you can delegate this to the Database Server. The only work you have to do is to specify the expiry date and MongoDB takes care of it automatically.

You can read more about this functionality here 👇🏻👇🏻👇🏻

Expire Data from Collections by Setting TTL - MongoDB Manual

Consider another scenario where you have to store logs for the last 7 days and keep deleting the old data and if the logs exceed storage limits, the logs need to overwrite the old data to store new data.

Can you guess a general flow for this?

This is how you need to do if you are using MySQL. But if you are using MongoDB, you can use the combination of “Capped Collections” and “TTL” to do all this for you without writing a single line of code.

Capped Collections:

Capped collections are fixed-size collections that support high-throughput operations that insert and retrieve documents based on insertion order. Capped collections work in a way similar to circular buffers: once a collection fills its allocated space, it makes room for new documents by overwriting the oldest documents in the collection.

Insertion Order:

Capped collections guarantee preservation of the insertion order. As a result, queries do not need an index to return documents in insertion order. Without this indexing overhead, capped collections can support higher insertion throughput.

Automatic Removal of Oldest Documents:

To make room for new documents, capped collections automatically remove the oldest documents in the collection without requiring scripts or explicit remove operations.

Consider the following potential use cases for capped collections:

Store log information generated by high-volume systems. Inserting documents in a capped collection without an index is close to the speed of writing log information directly to a file system. Furthermore, the built-in first-in-first-out property maintains the order of events, while managing storage use.

Cache small amounts of data in a capped collections. Since caches are read rather than write heavy, you would either need to ensure that this collection always remains in the working set (i.e. in RAM) or accept some write penalty for the required index or indexes.

• Capped Collections
• Expire Data from Collections by Setting TTL - MongoDB Manual

Let’s talk about the overheads that everyone keeps mentioning without any explanation

MySQL has more overhead than MongoDB. If you still haven’t figured that for yourself by now, read the article again.

Most of the Software Engineers whom I interviewed always say one thing about why MongoDB is faster than MySQL

If you are a developer, then most probably you know that the most of the CPU time is wasted on waiting for I/O. So, most of them I interviewed gave me the same answer: “MongoDB performs faster I/O and that’s why it is faster than MySQL”

The thing most of them don’t realise is, both of them are open-source projects (GPL) and if MongoDB is implementing a new faster strategy for I/O, then nothing is stopping MySQL team from pulling that code and pasting it in MySQL.

So, why insertions are slower in MySQL than in MongoDB?

Why MySQL is slower than MongoDB is because: MySQL has to check for a lot of rules before inserting, deleting and updating the data. This slows down the MySQL a lot for insertions. Look at the above flowcharts for yourself. For MongoDB, we are doing all the checks explicitly and then inserting into the database. Most of the benchmarks present on the internet doesn’t log the time during the checks we were performing. They only log the time needed for insertions.

But in MySQL, the database server performs all the checks implicitly based on our constraints and foreign keys before inserting into the database. And if you haven’t guessed it, benchmarks measure all these timings.

In MongoDB, an insertion command → Inserts the data

In MySQL, an insertion command → Check all the rules and insert only if it satisfies all those rules.

This is why MySQL has more overhead than MongoDB and effects the insertion throughput.

If you actually log only the time taken to insert into the databases with more complex relations like in real world instead of dumping data, you see that both are not so apart in benchmarks.

So, why retrieving data is slower in MySQL than in MongoDB?

In MongoDB, all the data of an entity is stored in a single document. So, if you want a particular data about a certain entity, then MongoDB uses Indexes to find the location of the document in the filesystem and retrieves that data. The document is stored in one location, so it is easy to retrieve that in one go.

In MySQL, all the data of an entity is stored in different tables. So, if you want a particular data about a certain entity, then MySQL uses many Indexes to find the location of the record in the filesystem and retrieves that data which is scattered around the filesystem and requires to visit different parts of the filesystem.

This slows down the MySQL greatly when compared to MongoDB.

MySQL vs MongoDB 1000 reads

So, what’s the summary?

MySQL is not slower than MongoDB without any advantages and MongoDB is not the database to choose for every project just because you see a bunch of articles claiming MongoDB is faster than MySQL. You also need to look at why MySQL is slower than MongoDB at what costs.

Let me summarise for you for the last time:

MySQL is slower because of the normalization, foreign keys, constraints, associations(relationships between tables) etc but reduces programmer bugs to maintain data integrity perfectly and MySQL does all the heavy lifting behind the scenes. But MySQL requires heavy optimisation from your side to perform fast and efficiently.

MongoDB is faster because there is no normalization(everything is stored in one place), foreign keys, associations etc but the programmer has to take care to maintain the data integrity. And to make sure performance doesn’t degrade with large data, you have to denormalize the data and store the same data in multiple collections or documents and if you want to change the data in one location, you have to programatically update all the locations(if you miss any location by chance, then your data will be inconsistent and data integrity is 💨💨) and let’s not forget about the increase in storage size.

A suggestion in choosing your next database for your personal project

Honestly, it doesn’t matter. The traffic you are expecting is too small to notice any difference between MySQL or MongoDB.

Remember, Twitter and Facebook still uses MySQL for their parts of databases. If they can use it, then you can use it without worrying about the performance in the initial stages of your projects.

So, don’t stress on which one to use. The more important thing is your IDEA. Focus more on that than on choosing types of servers, frontend, backend and database.

Just use whatever you know in your project. If you need an improvement to meet new traffic, then you will be obviously in a position to hire someone to do it from the revenue you get from the traffic.

So, don’t stress. Use whatever you know and improve it with time when the need comes.

Liked it? Then 👏🏻👏🏻👏🏻 it and share it!

SQL vs NoSQL: A Senior Software Engineer’s perspective was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.

First things first, this is not an another “KNOW THESE GIT COMMANDS TO BE MORE PRODUCTIVE!” or “GIT COMMANDS LIST THAT EVERY DEVELOPER MUST KNOW!”

We got enough of those in the internet. What we don’t have is: “COMMANDS YOU NEED TO KNOW WHEN YOU ARE MESSED UP!”😇

If anyone say they never messed up in Git, Don’t trust them! Unless it’s Linus Torvalds or Jeff Dean.

Git is every junior developer’s nightmare when they are working in a group. Especially with the people who are experienced and know their shit. Because,

1. If you make some mistake, you can’t just delete .git from the root folder and pretend everything is ok. (Ohhhhh… I’m guilty of doing this🤫. I hope my boss doesn’t read this article😰)
2. When they see your mess, they might not think you are dumb or inept of doing work but your conscious will think about it constantly. For people like me, the brain suddenly remembers all the embarrassing things even done during the childhood at 2 A.M.

The above two points makes everyone nervous before making every commit and pushing to the remote.

But sometimes, that extra care we take won’t even save us and sooner or later we do something stupid, which we want to hide desperately from our peers and manager. So, obviously we turn to Google, frantically searching for some commands for hours to bury the body and pretend everything is normal.

Ok. Ok. Fine😑. I’m gonna stop blabbering and get to the stuff you are here for before you feel sleepy🥱 and leave 🏃🏻‍♀️💨.(Sometimes, I get distracted 😅. I’m known to talk about ………….. See, I’m again getting distracted 🤦🏻🤷🏼)

Actually, it’s a two part series. I don’t want to fry your brains with too much information in one day(Yes, I do care for you🥺). So, this part mainly concentrates on commands which has effects on your local working directory. The next part will be on remote repo like on reverting, deleting, editing commits etc. which requires more of your attention while learning.

What’s in this?

1. Accidentally staged files for commit?
2. Accidentally tracking files or don’t want to track a file anymore?
3. Want to delete untracked and ignored files?
4. Want to revert un-committed changes?

So, let’s see some of the things we can mess up and how to clean it without jeopardizing our whole work.

Accidentally staged files for commit?

Have you ever executed the below command and wanted to undo?

git add .

git add *

git add -A

git add -u

(Yes, there are 4 variations! Checkout this SO post to know everything about them)

git add * (asterisk) vs git add . (period)

Now, you got multiple commands to do that

1. git reset <filename> or git reset HEAD <filename>
2. git rm --cached <filename>
3. git restore --staged <filename> — Added in Git v2.23 (August 2019)

Actually, these three commands will be recommended to you by the Git itself when you add the files to index or to staging area. If you are using Git v2.23+ then you will be suggested git restore --staged <filename> instead of git reset HEAD <filename>

I guess the new command git restore --staged <filename> is self explanatory. Let me explain the other two commands.

git reset HEAD <filename> : reset command resets the INDEX or STAGING AREA to previous commit i.e., the position of the HEAD in the Git.

git rm --cached <filename> : removes the files from the INDEX or STAGING AREA completely. If you deleted a file which is already in the repo and run this command, the file will be removed from the INDEX or STAGING AREA and also from the repo in the next commit. That said, if you used git rm --cached on a new file that is staged, it would basically look like you had just un-staged it since it had never been committed before.

Now, why does Git suggest two different commands?

For git reset HEAD <filename> or git restore --staged <filename> to work, there should be at least 1 commit in the repo. Because git reset HEAD command is saying to reset the files to the HEAD but HEAD is not created until the first commit is made.

But you can still run it by excluding Head option and it works perfectly

WHY🤔?

In old versions of Git, the above commands are equivalent to git reset HEAD <file> and git reset HEAD respectively, and will fail if HEAD is undefined (because you haven't yet made any commits in your repository) or ambiguous (because you created a branch called HEAD, which is a stupid thing that you shouldn't do). This was changed in Git 1.8.2, though, so in modern versions of Git you can use the commands above even prior to making your first commit:

“git reset” (without options or parameters) used to error out when you do not have any commits in your history, but it now gives you an empty index (to match non-existent commit you are not even on).

Note: I will use git revert instead of git revert Head in the following article.

You can read more about git revert vs git revert HEAD in the below SO posts

• How do I undo 'git add' before commit?
• git reset vs git reset HEAD

So, if you observe next time, Git suggests git rm --cached <filename> when you don’t have initial commit and when you have at least one commit, Git suggests git reset HEAD or git restore --staged

Now, you might be scratching your head thinking: “I understand everything but still I don’t understand anything and how to use it”. Don’t worry, been there, done that. So, how about we look at those commands from a different perspective with a mess up to understand it actually.

git reset

git reset HEAD <filename> : You are lazy and you commit only at the end of the day by staging files and committing multiple times to organise the work.(Yeah… Sometimes I do this too😑)

Entire day you have added/edited many different files like above. Now you want to group server config files in one commit and database config files in one commit. But accidentally you have executed git add .

Now, you see that and want to remove the db config files and any other files to commit only server config files

Now, you can commit server config files and then commit db config files in separate commits.

Doing this doesn’t delete anything from your local working directory.

git rm — cached

git rm --cached : Now you see why Git suggests this only when there are no commits. Tip: Don’t use this to unstage files without knowing completely about this!

I repeat! The behaviour of git rm --cached and git reset is same when there are no commits!

First, let’s see what is the output when we run git rm --cached with at least one commit

Can you comprehend what happened by just looking at the above image? If not, don’t worry. That’s why everyone is here. Nobody is gonna judge you.

git rm --cached deletes the file from the INDEX or STAGING AREA but you still have the file in your filesystem with all it’s content and also Git removes it from it’s tracking.

In simple terms, git rm --cached makes it look like you copied a new file into your folder.

Another effect is, this command removes the file from the repo. If anyone pulls the code after you have run git rm --cached <file> && git push , then that <file> will also be deleted from their local filesystem. Because it only exists in your local filesystem as an untracked file.

I’m gonna give you an embarrassing mess once I made so that you can get an intuition when to use this.

Story Time: Once, I initialised my node project and forgot to add .gitignore and ran my alias to add everything and push it to the remote server.(Yeah I know how dumb it is and my brain is known to glitch out sometimes) Doing that pushed my entire node_modules to the remote server. So, what can I do?

1. Add .gitignore, run rm -r node_modules && git add . && git commit && git push && npm install OR
2. Add .gitignore, run git rm --cached -r node_modules && git commit && git push

The second option is more nice and short. When you run git rm --cached -r ( -r stands for recursive)it removes the node_modules from the INDEX or STAGING AREA and also removes it from it’s tracking files but it is not removed from your local file system. So, you don’t have to install it again like in first option. Now, whenever you run git add . , it checks .gitignore and you won’t see node_modules in your git status as untracked files.

Doing this doesn’t delete anything from your local working directory.

Summary:

Always use git reset <filename> to unstage some files or git reset . to unstage all files. This works even if there was no prior commit in the repo.

Use git rm --cached only when you want to remove the file from the repo but keep a local copy in your local file system. Don’t use it interchangeably with git reset !

If you want to know from more why there are two ways, checkout this SO post:

Why are there two ways to unstage a file in Git?

Accidentally tracking files or don’t want to track a file anymore?

This is where git rm --cached is useful actually.

Like I said before, (read the Story Time above if you skipped it to understand)

Ok Fine. I’m gonna say it again in a summary what it does for some lazy people.

git rm --cached command actually removes it from the INDEX or STAGING AREA and also deletes it from the repo. So, git push actually removes the file from the repo and any subsequent git pull by others will also delete that file from their local file systems(not yours. You actually have your copy).

To remove/untrack everything: git rm --cached .

To remove/untrack one file: git rm --cached <filename>

To remove/untrack a directory: git rm --cached -r <directoryname>

Doing this doesn’t delete anything from your local working directory.

Note: git rm is not same as git rm --cached . git rm actually deletes the file from your local file system, repo, and others file system. So, be careful not to forget — -cachedoption!

How to make Git "forget" about a file that was tracked but is now in .gitignore?

Want to delete untracked and ignored files?

Suppose you added some new files to the working folder and later decided to remove those. There are too many files to run rm file1 file2 file3 .... .

Another scenario is, you want to delete the temporary files which are mentioned in .gitignore

Doing this deletes from your local working directory.

Be careful!

You can run git clean

From man pages:

Cleans the working tree by recursively removing files that are not under version control, starting from the current directory.

Normally, only files unknown to Git are removed, but if the -x option is specified, ignored files are also removed. This can, for example, be useful to
remove all build products.

As said, it removes recursively from current directory(where you run that command). So, to remove all untracked and/or ignored files/directories, run it from the root of the project directory.

To remove untracked files: git clean -f

To remove untracked and ignored files: git clean -xf

To remove untracked directories and files: git clean -fd

To remove untracked and ignored directories and files: git clean -xfd

Why -f, --force?

If the Git configuration variable clean.requireForce is not set to false, git clean will refuse to delete files or directories unless given -f, -n or
-i. Git will refuse to delete directories with .git sub directory or file unless a second -f is given.

This command actually deletes files and directories physically. So, you might be intimidated by running this command and hope you are not doing wrong by deleting “not-to-delete” files/directories.

If you are scared(like me), you can use this command in “Interactive Mode” by running git clean -fi

You can select option 4 and Git asks whether to remove it or not for every file.

From man pages:

Want to revert un-committed changes?

You have run git commit and later edited some files in the project. After hours of work, you want to delete the changes or un-edit those files to previous staging version.

You can use: git checkout

Yeah. You heard that right. The innocent git checkout which we often use to change branches can actually cause some destruction if used improperly.

Doing this deletes the work from your local working directory up until previous commit.

Be careful!

From man pages:

git-checkout — Switch branches or restore working tree files

Updates files in the working tree to match the version in the index or the specified tree. If no paths are given, git checkout will also update HEAD to set
the specified branch as the current branch.

Just a side note: You might have seen git restore <filename> as a suggestion by Git in the above image(that might be because I’m using latest version v2.25.0). Don’t use that because it is experimental as it is mentioned in the man pages

Moving on ….

git checkout . can only revert any modified file to previous staging version but not to previous commit!

In the above image, the previous staging version of somedbconfigfile1 is empty. So doing git checkout . reverted the contents First message to commit to empty.

But you might think “In the above image, git checkout .reverted to previous commit right?”

Now, let’s see another example to really understand what reverting to previous staging version means.

Git doesn’t keep history of INDEX or STAGING AREA. It only has two states.

1. What’s there in INDEX
2. What’s not there in INDEX

If you run git add , Git replaces/adds it’s “What’s there in INDEX” with new “What’s not there in INDEX” data. So, you cannot iterate over your INDEX history like with your COMMIT history and restore.

So, as there are no INDEX versions left, Git gets the copy from previous commit which is somedbconfigfile1 is empty.

git checkout <filename> command can only be run on un-staged files. That’s the reason I have used git reset . to unstage the file in the above image.

I know that’s a lot of things to absorb about the innocent git checkout .

I think it’s enough for today. Don’t forget to comeback to check the part 2! If your memory is as bad as mine, Hit that FOLLOW 😉

Other useful articles

If you still got some energy to burn, then read this one about git reset. It is a must! Complete guide for git reset : https://git-scm.com/book/en/v2/Git-Tools-Reset-Demystified

If anyone finds another way to do the same things with different commands, Congratulations! Do share and enlighten our fellow developers! If you don’t know then yes, we can do same thing using different commands in Git. Don’t trust me? Look 👇🏻👇🏻👇🏻

Disclaimer: Before anyone calls me out for not giving credits for memes, I have a dope stash of them saved on my phone from all over the internet since the “dot com bubble”. So, if you need some, mail me🙂

Liked it? Then 👏🏻👏🏻👏🏻 it and share it!

Git: Commands you need to know when you are messed up😨! was originally published in From The Scratch on Medium, where people are continuing the conversation by highlighting and responding to this story.