Why Your TV Makes Movies Look Like Soap Operas
Zoom in far enough on any digital image and the illusion breaks down into squares. Video Forensics #2: pixels, sampling, resolution, and frame rate, the building blocks of every digital video
Before I can get into source identification or tamper detection in this series, I need the actual vocabulary nailed down. What a digital video is made of, how it gets sampled, how frame rate actually works. This is the building blocks post, episode two.
Digital Videos: Basic Concepts
Before I can get into source identification or tamper detection in this series, I need the actual vocabulary nailed down. What a digital video is made of, how it gets sampled, how frame rate actually works. This is the building blocks post, episode two.
What a Digital Video Actually Is
A digital video is a time varying sequence of images. A series of digital images, called frames, displayed one after another in rapid succession. That’s different from analog video, which represents moving visual images using an electrical signal captured onto magnetic tape. Digital signals are discrete time signals generated through digital modulation. Analog signals are continuous signals representing physical measurements directly. The core difference comes down to discrete versus continuous representation of the same visual information.
Pixels and the Layout of a Frame
A single digital frame is made up of rows and columns of small single colored square blocks called picture elements, or pixels. Under normal viewing conditions pixels aren’t discernible to the human eye. Enlarge a digital image or frame enough and they become visible. That process of zooming in until individual pixels show up is called pixelation, which is exactly what’s happening in the header image above.
Mathematically a 2D video frame is represented as I(m,n), where m and n are the spatial coordinates designating the row number and column number in an MxN frame, and I is the intensity or color value of the pixel sitting in the mth row and nth column. Intensity is represented as a positive integer.
How a Digital Video Gets Created
Information in the natural world, the kind we take in through our five senses, is analog. Infinitely variable. Digital audio and visual information instead consists of discrete units of data placed close enough together that our senses perceive them as a continuous flow.
A natural visual scene is spatially and temporally continuous. For a computer to represent that scene it first has to be digitized, converted into a stream of zeros and ones. Digitization requires two operations: sampling and quantization. Sampling means taking measurements at discrete locations to convert a continuous time signal into a discrete time signal, and it has to happen in two dimensions for video, spatially and temporally.
Spatial Sampling and Spatial Resolution
Spatial sampling means taking measurements of color at discrete locations within the frame. The most common format is a rectangle with sampling points positioned on a square or rectangular grid, sampling at each intersection point, with the frame reconstructed by representing each sample as a square pixel.
Choosing a coarse sampling grid with fewer points produces a low resolution frame. Increasing the number of points increases resolution.
Spatial resolution, also called pixel resolution, is defined in terms of a video’s horizontal and vertical resolution and relates to pixel sampling density, the number of pixels per unit distance. It’s commonly quoted as C x R, the number of pixels in the column and row dimensions, for example 640x480, 800x600, 1024x768.
A few standard formats worth knowing: SQCIF (Sub Quarter CIF) at 128x96, QCIF (Quarter CIF) at 176x144, CIF (Common Intermediate Format) at 352x288, and 4CIF at 704x576. The choice depends on the application and available storage or transmission capacity. 4CIF suits standard definition TV and DVD video, CIF and QCIF suit video conferencing, QCIF or SQCIF suit mobile multimedia where display resolution and bit rate are limited.
Display devices like TVs and monitors have their own standard display resolutions too, tied to the number of distinct pixels they can show. The ratios that come up alongside these, 4:3, 5:4, 16:9, are aspect ratios, the proportional relationship between width and height. The term applies to video frames and digital images the same way it applies to screens.
Temporal Sampling and Frame Rate
A digital video is basically a sequence of still frames, each one a snapshot taken at a particular moment in time, captured at periodic intervals to produce the appearance of motion on playback.
Get Awais Ahmed’s stories in your inbox
Join Medium for free to get updates from this writer.
Just as spatial sampling relates to spatial resolution, temporal sampling relates to temporal resolution, the number of frames captured in a given time period. Temporal resolution depends on frame rate, denoted in frames per second (fps), the rate at which consecutive frames are displayed. Higher frame rate, better temporal resolution.
Frame Rate and Human Vision
The human eye can differentiate between 10 and 12 still frames per second before it starts perceiving continuous motion. Below 12 FPS, the visual system perceives each frame individually and motion looks unnatural or choppy, sometimes called flickering. Between 12 and 20 FPS motion looks smoother but only in the slow moving parts of a sequence. Above 20 FPS, motion looks smoother overall. 25 or 30 FPS is standard for television, usually with interlacing. PAL runs at 25 FPS, NTSC runs at 29.97 FPS.
Recording frame rate and playback frame rate can differ, and that difference affects perceived motion. Equal rates look like real time. A lower recording rate than playback, say recorded at 60 FPS and played at 30, looks like slow motion. A higher recording rate than playback looks like fast motion.
The Soap Opera Effect
This is the part that actually explains something I’d noticed for years but never had a name for. It’s why movies and TV shows feel like different visual experiences, to the point you can often tell which is which from a ten second clip. It’s called the soap opera effect, and it comes down to frame rate.
Movies and primetime dramas are traditionally shot on film at 24 FPS. Ordinary TV shows, soap operas, sports, reality TV, are shot on video at 30 or 60 FPS, which gives them that smooth, glossy, almost too real look. It’s also the same effect modern HD TVs introduce through motion smoothing features, which end up making a cinematic movie look like a made for TV movie instead.
It’s why the 3D high frame rate version of The Hobbit: An Unexpected Journey felt off to a lot of people. Shot at 48 FPS, twice the conventional 24 FPS standard, the higher frame rate produced motion so smooth it looked unnaturally crisp, like someone had just followed the actors around with a camcorder rather than filming a stylized movie.
Temporal Interpolation and Frame Rate Conversion
Sometimes it’s desirable to change a video’s frame rate after it’s already been recorded. This is post production frame rate conversion (FRC), increasing (up converting) or decreasing (down converting) frame rate. Frame rate up conversion (FRUC) is the more common of the two, done by periodically inserting newly generated frames between original frames through frame interpolation or frame blending, estimating new frames using existing, temporally neighboring ones.
FRUC techniques split into two categories: simple techniques that ignore object motion (frame repetition, frame averaging), and object motion based techniques that use motion compensation for better quality (Motion Compensated Frame Interpolation).
- Frame repetition. The simplest approach. A particular frame is simply repeated in between existing frames, and the process repeats across the entire sequence.
- Frame averaging. New frames are constructed by linearly averaging a set of frames within a pre-defined temporal window.
The worked example from the lecture makes frame averaging concrete:
The top right pixel goes from 12 in the first frame and 55 in the second to (12+55)/2 = 33.5, rounded to 34 in the new frame. The bottom left pixel goes from 55 and 103 to (55+103)/2 = 79. A straightforward linear average applied pixel by pixel across corresponding positions.
Both frame repetition and frame averaging fail to produce smooth motion in the up-converted sequence. Object or camera motion still looks jumpy, because neither technique accounts for where things in the frame are actually moving to.
Motion Compensated Frame Interpolation (MCFI) uses motion compensation to estimate new frames from the motion information between adjacent frames, detecting how an object or the camera moves between successive frames and using that to generate intermediary frames that preserve fluid motion. Videos up-converted this way generally don’t suffer from jerky object motion, but the perceived smoothness can itself trigger the soap opera effect, since it’s the same underlying mechanism modern TVs use for motion smoothing.
Where This Series Goes From Here
Next up is getting into actual source identification and tamper detection techniques, now that the vocabulary for how a digital video is built is in place.
If you’ve got questions on any of this, reach out through LinkedIn.
Thanks!
