What’s a Buffer?
Table of Contents
· Introduction
· Binary Numbers
· Hexadecimal Numbers
· Character Sets
· Buffer
· Conclusion
· References
Introduction
I frequently encountered the use of “Buffer”. Especially when I develop in Node.js environment. For example, we often see Buffer.from() being used to handle binary data.
So why do we use “Buffer” instead of just using regular Javascript Array or Strings ? It is because it helps us to deal with ‘0’s and ‘1’s directly. It connects JavaScript or any other programing language to these binary data which allows direct manipulation and handling of individual bits during the development process.
Before diving into the concept of “buffer”, It’s essential to have strong foundation of binary number, hexadecimal number, character sets. Therefore, I am going to divide this article into four sections :
- Binary Numbers
- Hexadecimal Numbers
- Character Sets
- Buffers
Binary Numbers
The CPU in computer can understand and process binary numbers using transistors , which manipulate binary data
The CPU in computer operates by understanding binary numbers. This is done by transistor. Transistor, acting as electronic switches, are instrumental in manipulating these binary bits.
Each transistor can be
- ON ( electricity flows ) => represents 1
- OFF ( no electricity ) => represents 0
They form logic gates, enabling the CPU to perform complex operations, store and retrieve data.
The table above shows transistor’s count in each of Apple’s chips. The M2 Max even has 67 billion transistors ! It shows how powerful it is.
This two-state system (ON/OFF) is binary system, which forms the foundation of how computers represent and process data.
Now let’s explore how binary numbers work. Suppose we have 8 binary numbers which is 0000 1011. Each single number whether it is 0 or 1 , represents “1 bit”, the most basic unit in computer. The rightmost bit is called LSB(Least Significant Bit), and the leftmost bit is called the MSB(Most Significant Bit). When the bits are grouped into sets of 8, we call it “byte”. So basically, 1 byte is a group of 8 bits.
When you read binary numbers, you need to read from the rightmost side which will be the index 0, and to interpret it into decimal number, multiply each 1 or 0 by 2 to the power of based on its index. So 1011 will be calculated as follows.
1 * 2⁰ = 1
1 * 2¹ = 2
0 * 2² = 0
1 * 2³ = 8
At the end , we add all of the results into one which will be calculated as 1+2+0+8 = 11. So 1011 is 11 in decimal number.
Hexadecimal Numbers
Every 4 bits in bytes can be represented by a single hexadecimal digit.
As a software engineer, understanding hexadecimal numbers is crucial due to their frequent use in computer. This preferences arises from the simplicity of converting binary numbers to hexadecimal, where every 4 bits in bytes can be represented by a single hexadecimal digit.
For instance, if we have a file consisting of 4000 bits using binary numbers, converting to hexadecimal characters allows us to interpret it with just 1000 bits. Computers often utilize hexadecimal numbers to reduce the size of data representation.
The picture above shows hexadecimal number, since hexadecimal number uses 16 characters, if it uses only decimal numbers like 0 to 9, it can be confusing that if the number is decimal or hexadecimal. That’s why it uses so called “indicator “ to show that the number is hexadecimal. The “0x” is a indicator that the number is hexadecimal.
The calculation of hexadecimal to decimal is similar to the binary number. Multiply each numbers by 16 to the power of based on index. 0x235 will be calculated as follows.
5 * 16⁰ = 5
3 * 16¹ = 48
2 * 16² = 512
And if we add all the result , 5 + 48 + 512 will be 565 in decimal.
The number system overall look as follows.
From 0 to 9 are same numbers as decimal, But from A to F are new numbers which is 10 to 15 in decimal. The characters are case insensitive.
Get bytePudding’s stories in your inbox
Join Medium for free to get updates from this writer.
Let’s say we want to calculate a2f to decimal. The calculation will be operated as below.
f : 15 * 16⁰ = 15
2 : 2 * 16¹ = 32
a : 10 * 16² = 2560
And if we add all the result , 15+ 32+ 2560 will be 2,607 in decimal.
There is even binary to hexadecimal charts. “0000” in binary number is equal to “0” in hexadecimal. similarly, “1010” in binary number is equal to “A”. If there is a memory location of 0001 1011 1100 in binary number, If it is written in hexadecimal number it will be “1BC” which is much simpler. In computers, data is stored in binary, but tools like debuggers or memory viewers often display the content in hexadecimal for convenience.
We can also calculate the hexadecimal number to binary using the default calculator in your laptop. If I just click “F”, It shows 15 in decimal, and 1111 in binary. It also means that If I input 15 decimal number, computer will convert it to 1111 in binary number and when we look at the memory content, computer will show F which is hexadecimal.
Let’s investigate this deeper. For example, my laptop’s battery is 98%.
If I convert 98 to binary , it is 0110 0010
So somewhere in my memory, it stores “0110 0010”.But as conventional in computer, If I want to see the content of the memory, computer will show hexadecimal number which is “0x62” to us.
Let’s play around with the calculator more. If I click 1, It will be 0001 in binary.
Let’s flip the zero to one in 40. It becomes huge numbers just by flipping it.
This shows that binary representation is well-suited for computer because it is faster and efficient than decimal number. (Though this is not the main reason why computer uses binary number. I won’t talk more on it as it’s out of topic)
Character Sets
Character sets are the mapper between human language and computer language
As computer understands only binary numbers, how can computer understand strings and characters? It is done by character sets which is a mapper between human language and computer language.
There are two character sets standard used worldwide, Unicode and ASCII. Unicode holds almost every languages worldwide such as Korean or German. It holds 149,813 characters in 15.1 version.
ASCII on the other hand, holds 128 characters because ASCII is focused on English letters. “A” in ASCII stands for “American”.
Let’s explore ASCII more. the picture above shows the list of ASCII characters. As you can notice, each character holds two numbers in hexadecimal number. Because 1 number in hexadecimal represents 4 bits, we can say that every characters in ASCII holds 8 bits which is 1 byte.
As ASCII is a subset of Unicode, Unicode holds ASCII characters with same binary number which is 1 byte. Therefore the charactor “A” has the value 65 in both ASCII and Unicode. However, other characters that are not part of ASCII can take up more space — such as 16 bits or even 32 bits depending on the encoding used in Unicode.
Unicode is widely used in computer’s character encoding system. “Encode” means converting something meaningful to human to binary code. For example if we encode “A” , it will be 0110 0001. It basically assign a sequence of bytes(0 and 1) to a character. Most commonly used character encoding is UTF-8 which uses Unicode character sets.
Buffer
Buffer is a memory container that holds data in binary form.
Buffer is a memory container that holds data in binary form. For example, we can allocate 2 bytes in the buffer . The default buffer will be look as follows
[buffer] 0000 0000 0000 0000
If we input something to a buffer, It will be changed as below
[buffer] 0000 0010 0100 0110
A buffer operates in binary, making operations faster. When allocating a byte size in the buffer, such as 2 bytes, it means that we are occupying 2 bytes, equivalent to 16 bits, in our memory.
As shown in the diagram above, buffers operate in bytes (binary numbers). When data is input from sources like keyboards, files, and networks, the resulting binary data (bytes) is processed and displayed through output channels like consoles, files, and networks.
To manipulate this raw binary data, for example, in JavaScript we use Buffer as code below
// create a buffer from string, Now we can manipulate as raw binary data !
const buffer = Buffer.from('Hello', 'utf-8');
// it shows binary data (in hexadecimal form)
console.log(buffer) // <Buffer 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21>
// let's manipulate this binary to a string !
const stringFromBuffer = buffer.toString('utf-8');
console.log(stringFromBuffer); // Hello! This is a simple example, but when you’re dealing with network packets, file I/O, or binary file formats — which often involve large amounts of data — buffers allow for efficient memory usage by working directly with the byte-level representation of that data.
Conclusion
A buffer is a memory container that operates in bytes(binary numbers). Since computers operate in binary number, unlike other data structures, buffer allows direct access to memory. When handling memory efficiently, buffers can be a useful tool due to their capability to directly interact with memory.
References
Joseph Heidari’s Understanding Node.js: Core Concepts on Udemy
