Raw HD Video Is 11 GB a Minute. Compression Fixes That by Exploiting Your Eyes
Fourth entry in my digital video forensics series. Episodes 2 and 3 covered the vocabulary and quality fundamentals, sampling, quantization, interlacing, camera categories. This one moves into compression: why a raw digital video is absurdly large, what “redundant” actually means in a video signal, and the three mechanisms, spatial, temporal, and coding, that compression exploits to shrink that signal down without the viewer noticing.
Why Compression Is Necessary
A typical HD video, 3 color planes, 1920×1080 resolution, 8 bits per pixel, recorded at 30 FPS, needs a data rate of approximately 1.5 billion bits per second. A one-minute clip at that rate is 11 GB. An hour is 670 GB. Uncompressed digital video is not a practical format for storage or transmission at any meaningful scale.
Fortunately, digital video signals carry a great deal of redundancy, information that can be eliminated without the viewer perceiving anything missing. Exploiting that redundancy is only possible once you understand the imperfections of the human visual system (HVS), because “redundant” ultimately means redundant to a human observer, not redundant in some abstract mathematical sense.
The Human Visual System and Why It Matters for Compression
The HVS converts light into neural signals through two types of photoreceptors. Rods are sensitive to low light, can’t distinguish color, and dominate the periphery, they’re responsible for motion and shape detection in peripheral vision. Cones are sensitive to higher light levels and form the basis of color perception, concentrated in the fovea, the central region of the retina. Rods outnumber cones by more than an order of magnitude, about 100 million rods versus 6.5 million cones, which is why the HVS is far more sensitive to motion and structure than to color or texture.
Color perception itself comes from three types of cone cells:
• A cones, sensitive to red light.
• B cones, sensitive to green light, with higher sensitivity than A, which is why human eyes are most sensitive to green.
• C cones, sensitive to blue light, at roughly 1/30th the sensitivity of A or B.
This is trichromatic vision. When one of the three cone types is dysfunctional, the result is a specific, named form of color blindness: protanomaly (reduced red sensitivity), deuteranomaly (reduced green sensitivity, the most common form), or tritanomaly (reduced blue sensitivity, extremely rare). If only one or none of the cone types is functional, the result is monochromacy, color vision collapsing to shades of black, gray, and white.
Once you understand the HVS’s characteristics, you can exploit its imperfections. The underlying principle behind every data reduction policy used in video compression is the same: eliminate any information the brain will not perceive as missing. Optical illusions are a direct demonstration of this exploitability, they occur when the visual information the brain perceives differs from objective reality, either because the brain fills in gaps in incomplete information, or because it’s overwhelmed with enough inane detail that it fails to process something hidden in plain sight.
Redundancy: The Core of Compression
Digital video signals contain a lot of similar, correlated information between neighboring pixels and neighboring frames. All of that is redundant information: it can be eliminated or reduced without affecting the brain’s perception of the video’s content, and that is the essence of compression. Formally, redundancy is defined as 1 minus the ratio of the minimum number of bits actually needed to represent an image to the number of bits currently used to represent it. Redundancy is higher in images with a lot of spatial detail, like foliage, and lower in images with fewer details, like a plain face.
Compression comes in two kinds. Lossy compression permanently discards some data, meaning the original can never be recovered exactly. Lossless compression can be fully undone, recovering the exact original, because no data is lost. Lossy compression can seem like a bad trade given the data loss involved, but the trick is to lose only the data that is unnecessary for accurate perception of the scene by a human observer. Depending on the algorithm, lossy compression may produce results visually indistinguishable from lossless, or it may cause significant, visible loss of detail.
Video redundancy breaks down into a clear taxonomy. Statistical redundancy is caused by correlation, either between pixels of the same frame (spatial redundancy) or between neighboring frames (temporal redundancy), together known as interpixel redundancy, or by how the information is represented (coding redundancy). Psychovisual redundancy is a separate category entirely, rooted in what the HVS simply doesn’t register.
Spatial Redundancy
Spatial redundancy is the consequence of correlation among neighboring pixels within a single frame, also known as intra-frame or intra-picture correlation. Adjacent pixels tend to have very similar intensity values, which means they’re largely redundant and don’t carry new information. Eliminating these redundant pixels can substantially cut the number of bits needed to represent a frame’s content.
How much spatial redundancy is present in a given frame depends on the sampling rate, number of quantization levels, and noise, higher sampling rates, fewer quantization levels, and less noise all mean higher pixel-to-pixel correlation, and so more exploitable spatial redundancy.
Temporal Redundancy and Frame Prediction
Temporal redundancy comes from correlation between adjacent frames in a sequence, inter-frame or inter-picture correlation. For a human observer to perceive smooth, continuous motion, video needs to be displayed at more than 15 FPS, which in turn requires neighboring frames to be very similar to one another.
Since so little actually changes frame to frame, it’s tempting to just reduce the frame rate to save data, but that comes at the cost of perceptible flickering. A more sensible approach is to represent a frame in terms of a neighboring reference frame plus the difference between the two. Since that difference information is much smaller than an entire frame, higher compression becomes possible by “predicting” frames from their neighbors. The reference frame that enables this prediction is called the independent frame; the frame being reconstructed with its help is the dependent frame.
Get Awais Ahmed’s stories in your inbox
Join Medium for free to get updates from this writer.
When very little has changed between two frames, the difference frame is mostly black, representing near-zero information, since subtracting nearly-identical pixels from each other yields values close to zero.
When there’s been more movement between the two source frames, the difference frame carries noticeably more visible information, faint but legible outlines of the moving subjects.
Putting it together: an independent frame plus its difference information reconstructs the predicted (dependent) frame.
The numbers make the case for this approach directly. In one worked example, two 700 KB frames total 1,400 KB stored independently. With prediction, the pair of frames plus a 25 KB difference frame comes to only 725 KB, a 48% reduction for just two frames. Scaled across an entire video, frame prediction (and, combined with motion compensation, even more so) is one of the biggest levers video compression has.
Coding Redundancy
All digital information is represented as a bit-stream, and more complex information means a longer bit-stream. A single 320×240 color video frame, 76,800 pixels at 24 bits each, requires 1,843,200 bits, over 1.8 million individual 0s and 1s for one frame alone. Coding redundancy is addressed by assigning every bit-stream a code, a system of symbols (letters, numbers, bits) that represents a body of information using far less data than the original. Codes do to information what abbreviations do to words.
• Run-length coding. Exploits runs of identical consecutive symbols. A binary stream like 1110 0000 0000 0000 0000 0011 can be coded as (1,3), (0,19), (1,2): three 1s, nineteen 0s, two 1s. Code and store the count of consecutive identical elements instead of the elements themselves, and the data reduction follows.
• Huffman coding. A probability-based strategy where each source symbol is encoded according to its estimated probability of occurrence. The most frequent symbols get the shortest codes; the least frequent get the longest.
Transform Coding
Spatial redundancy between neighboring pixels in a frame block isn’t, by itself, enough to make that block a good candidate for compression with variable-length coding techniques like run-length or Huffman coding. To make pixel blocks more amenable to that kind of compression, they first need to be represented in the transform domain, frequency components, rather than spatial dimensions. In the spatial domain a frame is rows and columns of pixel intensities; in the transform (frequency) domain, it’s a collection of transform coefficients.
Low-frequency components carry more of an image’s actual information than high-frequency components do, and there happen to be fewer low-frequency coefficients than high-frequency ones. That means the bulk of an image’s information is carried by a small handful of coefficients, while the majority of coefficients contain negligible information and can be discarded, zeroed out through quantization, before coding. Only the non-zero coefficients then need to be transmitted, which is where the real data reduction comes from.
The most prevalent transforms for this are block-based: the Karhunen–Loève Transform (KLT), Singular Value Decomposition (SVD), and the ever-popular Discrete Cosine Transform (DCT), which process the frame in small n×n blocks. The Discrete Wavelet Transform (DWT) is the most common image-based transform instead, processing the entire frame or a large section of it at once. Each applies a reversible linear transform meant to de-correlate the original signal and redistribute its information across a small set of coefficients, setting up the coding-redundancy techniques covered above to do the rest of the work.
Where This Series Goes From Here
With redundancy, frame prediction, and transform coding on the table, the series now has the conceptual groundwork for the compression standards that actually get used in practice, and for understanding compression artifacts as a forensic signal in their own right.
If you’ve got questions on any of this or want to compare notes, reach out. Website · GitHub · LinkedIn
Thanks!
