Deep Learning
Computing Neural
Network Data
Artificial Intelligence

4-Bit Deep Learning

Ditmonir

13 min readDec 14, 2020

Powerful neural networks can be trained on smartphones with dramatically faster speeds and less energy.

Bit needs Energy

Deep Learning is a very ineffective energy consuming process, because it requires large amount of data and ample of computational resources which is the reason for more energy consumption. This problem has become worse with the recent development to obtain a accurate model. The models have become larger and been trained on huge data points for days and will not stop since the data might appear in a new format which needs to be again trained.

Deep learning has a terrible carbon footprint.

There has been researches to who are trying to find algorithms that can use less data for training and hardware that can run the algorithm faster. IBM researchers are proposing on reducing the number of bits needed to represent the data to 4-bit size, which was presented in the NeurIPS the largest annual AI research conference, could increase the speed and cut the energy costs needed to train the deep learning model.

This also helps in training the powerful AI models possible on the smartphones and smaller computing devices, which helps in keeping the data local on the device and improving privacy for the end customer.

The computers stores and process data in the form of 1s and os and this fundamental unit of data is known as bits. This bits are combined into multiple strings of bits to represent the complete information. Two bits can represent four ways of representing the information because there are ²² combinations: 00, 01, 10, and 11. Four bits can represent ²⁴, or 16 ways of information. Eight bits can represent ²⁸, or 256. And so on.

In a 4-bit deep learning the 4-bit computer is able to represent the data using 4-bits. The data in the training process should be one of the 16 numbers between -8 and 7, since this are the only numbers the computer can represent. This is the same size for all the data provided to the neural network, used to represent the neural network, and the intermediate data to be stored during the training of the model.

To understand the above concept let us assume the training data is a black-and-white image. The image is converted to numbers for the computer to understand. Each pixel information is represented interms of the grayscale value 0 for black and 1 for white, the decimals can be used represent the shades of gray. the image is now a list of numbers between 0 and 1, which is represented between -8 and 7 in a 4-bit representation. The idea is to linearly scale the list of numbers, so value 0 becomes -8 and value 1 becomes 7, and the decimals map to the integers in the middle in the 4-bit representation.

Scale the number from 0 to 1 between -8 and 7, and then round any decimals to whole numbers

This may not be a accurate method since the value of 0.1 is -6.5 and would end up in representing a whole number by rounding -6.5 to -7. In this way you might be losing some precision in representing the image

Lower the number of bits the lesser the details of the photo

The neural network shown below contains nodes and connections. In a computer this turns into a series of numbers. Each of the node contains the activation value, which can range from 0 to 1 and the connections are called weight which can range from -1 to 1.

Neural Network

Get Ditmonir’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The weights and activations are scaled in the same way has pixels, but the activations and weights also change every round of the training. The activation ranges can be 0.2 to 0.9 in one round and 0.1 to 0.7 in the next round. In order to accommodate this changes the IBM group proposed to rescale those ranges to stretch between -8 and 7 in every round (as shown below), which effectively avoids losing too much precision.

Rescaling the activations and weights in the Neural Network

In order to represent the intermediate data that can grow during the training, the values can grow across several orders of magnitude, which is not like the values used to represent the images. The data can be either 0.001, or large like 1,000. The linear scaling between -8 and 7 cannot be used since it loses the granularity at the tiny end of the scale.

Linearly scaling numbers that span several orders of magnitude would lack of precision

The IBM researchers achieved by logarithmically scaling the intermediate numbers. The logarithmic scale you might recognize, with a so-called “base” of 10, using only four bits of complexity. (The researchers instead use a base of 4, because trial and error showed that this worked best.) From the scale we can see how it lets encode both tiny and large numbers within the bit constraints.

Logarithmic scale with base 10

By using all this information the latest paper published by IBM researchers by running several experiments where they simulate 4-bit training for a variety of deep-learning models in computer vision, speech, and natural-language processing. The final outcomes shows a limited loss of accuracy in the models’ overall performance compared with 16-bit deep learning. The process is also more than seven times faster and seven times more energy efficient.

Conclusion

Still there is a long way to go before 4-bit Deep Learing are put in practice. To implement in the real world there needs to be new 4-bit hardware designed and developed. This would enable in the existing computing devices to run faster and reduce the battery consumption, so that it can be deployed on smaller computing devices. Many companies are switching the process of training the AI models, like speech-to-text and autocorrect systems, away from the cloud and onto user phones. This helps in the users’ privacy by keeping their data on their own phone while still using the device’s AI capabilities.

Training a single AI model can emit as much carbon as five cars in their lifetimes

https://twitter.com/ufc257_live

References

8-Bit Precision for Training Deep Learning Systems
Ultra-Low Precision 4-bit Training of Deep Neural Networks
Convolutional Neural Networks using Logarithmic Data Representation

4-Bit Deep Learning

Get Ditmonir’s stories in your inbox

Conclusion

References

Written by Ditmonir

No responses yet

More from Ditmonir

Rebecca O’Brien of Wellness by Rebecca: Getting An Upgrade; How Anyone Can Build Habits For…

Prioritize self-care and relaxation. When it comes to performance at work or in a sport it’s about balance. Check-in with yourself daily…

How To Watch Tim Hortons Brier 2021 Live Streaming Online

Recommended from Medium

What Wealthy People Do Before 9 AM That Broke People Do Never

A five-year study tracked 233 millionaires and 128 people struggling financially. The differences weren’t income. They weren’t lucky. There…

10 AI Tools Everyone Will Be Using in Six Months, But Almost Nobody’s Found Yet

You’re going to see these in someone’s toolkit next year and wonder why you didn’t hear about them first. Here’s your head start.

8 New AI Terms You Should Know to Be Ahead of 90% of People

A few more AI terms to know beyond tokens, prompts, and hallucinations.

The Darker Side of Growing Old

The parts of aging we won’t Talk About

This is Feynman’s Thinking Habit. It Made Him a Genius

Think 10x clearer with this simple mental trick.

Andrej Karpathy’s New suggestion- Ask AI output in ASD-STE100 format and not in Text format

A few days ago, Andrej Karpathy shared a perspective that immediately clicked with how my own daily workflow has been evolving. His premise…