How CPU Caches Boost Performance (and Why Developers Should Care)

In the start, all computer components were on almost the same speed, life was good, and the programs were simple.

CPU becomes faster!

Time flies, and the CPU becomes faster. When the CPU becomes faster, a latency issue arises because internal components are not up to the speed of the CPU. For example, mechanical hard drives were slow(positioning time, etc.), and OS uses the main memory to keep the program running into it. The main memory speed was better than hard-drive access, but the CPU still needed many cycles to fetch the data from the main memory.

Can we make the main memory faster?

The answer is yes, we can, but it will make PC machines more expensive and may not be very energy efficient.

Therefore, the computer designers devised another idea to minimize the CPU to main memory latency by adding the CPU local caches so the CPU doesn’t need to go to main memory every time.

An example of a three-level cache of my laptop.

As you may notice in the screenshot, there are three levels of caches: L1, L2 & L3. L1 is small and fastest; L2 is bigger than L1 but less than L3.

CPU first checks with the caches(L1, L2 & L3, respectively), and if it finds the data, it fetches it; otherwise, the CPU works with OS(software) to find and update the cache lines.
single processor