Understanding Node.js Buffers

A buffer is a space in memory (usually RAM) that temporarily stores binary data (also called raw data). Buffer is used to temporarily store binary data while it is being transferred from one place to another. Before we dive deeper into buffers lets understand what is binary data.

Binary data:

Binary data is simply a collection or set of 1s and 0s. To store or represent a piece of data, a computer needs to convert that data to its binary representation because the computer processor is made of transistors that are activated by ON(1) and OFF(0) signals. For example-10, 01 , 001, 1110, 00101011 are five different binaries (five different sets of 1s and 0s). Each 1 and 0 in a binary is called a bit. A Byte is a set of 8 bits — eight 1s and 0s.

To store any character (e.g. “H”) in binary, computer will first convert that character to a number (using Character Sets), then convert that number to its binary representation (using Character Encodings).

Character Sets:

Character Sets are already defined rules of what exact number represents each character. Character Set translates characters to numbers. Some popular Character Sets are Unicode, ASCII etc.

All modern browsers and operating systems use Unicode internally and convert all other encodings to Unicode for processing. Unicode is a Character Set where each character is represented by a unique number (called a code point). You can think of Unicode as a map of every possible character to a decimal number (+ve integers).

In Unicode, character “L” for example is represented by 76 in decimal number (which converts to 4C in hex number).

let num = "L".codePointAt(0);
console.log(num);

Output:

76

You can get a list of Unicode characters here.

Character Encodings:

Character Encodings are rules that define how numbers should be represented in binary. Character Encoding translates number into binary. It defines how many bits to use to represent the number. Some popular Character Encodings are UTF-8, UTF-16 etc.

UTF-8 is the most popular Character Encoding. UTF-8 stands for Unicode Transformation Format — 8-bit. UTF-8 states that characters (represented by numbers) should be encoded in Bytes i.e. 8 bits. UTF-8 uses 1, 2, 3 or 4 Bytes to represent a unicode code point (number), depending on the character to save memory. Numbers are converted into binary using base-2 numeral system.

For example, “A” should be encoded as 01000001 and not 00000000 00000000 00000000 01000001. “😀” should be stored in binary as 11110000 10011111 10011000 10000000 and 12 should be stored in binary as 00001100.

let obj = Buffer.from("😀").toJSON();
// obj is equal to {type: 'Buffer', data: [240, 159, 152, 128]}

let data0 = obj.data[0].toString(2); // base 2 means binary
let data1 = obj.data[1].toString(2);
let data2 = obj.data[2].toString(2);
let data3 = obj.data[3].toString(2);

console.log(data0, data1, data2, data3);

Output:

11110000 10011111 10011000 10000000

Buffer:

A buffer is a space in memory that is used to temporarily store binary data. The Buffer class in Node.js is used to perform operations on binary data. The Buffer class was introduced as part of the Node.js API to make it possible to manipulate or interact with streams of binary data.

There are 2 ways to create a buffer — using Buffer.alloc() and Buffer.from() methods.

Buffer.alloc():

The Buffer.alloc() static method creates a new empty buffer object of the specified size.

// creates a new empty buffer of size 5 Bytes
let buf = Buffer.alloc(5);
console.log(buf);

Output:

<Buffer 00 00 00 00 00>

Buffer.from():

The Buffer.from() static method creates a new buffer filled with the specified string, array, buffer or arrayBuffer.

// creates a new buffer filled with the provided data
let buf1 = Buffer.from("ABCDE"); // String
let buf2 = Buffer.from([1, 2, 3]); // array
let buf3 = Buffer.from(buf1); // buffer

console.log(buf1);
console.log(buf2);
console.log(buf3);

Output:

<Buffer 41 42 43 44 45>
<Buffer 01 02 03>
<Buffer 41 42 43 44 45>

The numbers 41, 42, 43, 44 and 45 in <Buffer 41 42 43 44 45> are in hex.

Get Deepak Kumar Singh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Buffers store a sequence of integers similar to arrays. Each integer in a buffer represents a Byte. Buffers index their data like arrays, starting at 0. We can use array notation on the buffer object to get an individual Byte. To access one Byte of a buffer, we pass the index or location of the Byte we want.

let buf = Buffer.from("ABCDE");
console.log(buf[2]); // 2nd index of the buffer
console.log(buf[8]); // prints undefined because 8 is incorrect index

Output:

67
undefined

The difference between arrays and buffers is arrays can be any type and can be resizable while Buffers only deal with binary data and are not resizable.

Buffer.toString():

The toString() method returns a string form of the buffer according to the specified encoding. If no encoding is given, utf8 is taken as default.

let buf = Buffer.from("abcd");
console.log(buf.toString());
console.log(buf.toString("utf8"));
console.log(buf.toString("binary"));
console.log(buf.toString("hex"));
console.log(buf.toString("base64"));

Output:

abcd
abcd
abcd
61626364
YWJjZA==

Buffer.toJSON():

The toJSON() method returns the JSON object of the buffer.

let buf = Buffer.from("abcd");
console.log(buf.toJSON());

Output:

{ type: 'Buffer', data: [ 97, 98, 99, 100 ] }

Buffer.write():

It is used to write data to the buffer.

let buf = Buffer.alloc(10);               // create a new empty buffer
buf.write("Hello");                       // Writing to the Buffer
console.log(buf.toString());

Output:

Hello

Another example:

let buf = Buffer.from("ABCDEFG");
buf.write("HelloWorld!");
console.log(buf.toString());

Output:

HelloWo

Buffer.byteLength():

The Buffer.byteLength() static method is used to find the size of the buffer in Bytes.

let buf = Buffer.from("ABCDEF");
console.log(Buffer.byteLength(buf));

Output:

6

Buffer.copy():

The copy() method copies data from one buffer object into another buffer object.

let buf1 = Buffer.from("HELLO");
let buf2 = Buffer.from("abcdefghijkl");

// copy data from buf1 to buf2
buf1.copy(buf2);
console.log(buf2.toString());

Output:

HELLOfghijkl

Buffer.isBuffer():

The Buffer.isBuffer() static method is used to check if an object is a buffer.

let obj = Buffer.from("abc");
console.log(Buffer.isBuffer(obj));

Output:

true

You can look for more here.