The file extensions such as .jpg, .png, .mp3, .mp4 come from the different codecs used each with their own virtues and vices.

Types of codecs

When it comes to digital data, or multimedia, these codecs can be roughly divided into three main categories

• Audio codecs
• Video codecs
• Image codecs

And there are various types of codecs under these categories, some are open source and some are licensed.

These codecs can again be divided into two sub-categories based on the type of compression they perform on the input media, namely, lossless and lossy. Lossless gives higher reconstruction quality at the cost of computation time, complexity and storage and lossy compression is cheaper in terms of storage but provides lower resolution.

The choice of codec depends on the needs and importance of the application. For instance, applications including sensitive and important minute data such as medical applications might opt for a lossless codec whereas applications like video streaming via service providers or video calls can be carried out through lossy codecs.

For the sake of sanity of readers, we are only focusing on video codecs in the following sections.

Video codecs and where are they used?

With the rise of streaming services and customer demands, video codecs have evolved in a way to provide utmost quality with as less latency as possible. One of the peak points in the history of media streaming was the shift from standard definition (SD) to HD (High definition), it might seem very minute but it was a huge leap in codecs and their architecture with developments of techniques such as motion estimation, entropy coding etc.

A list of popular services with their codecs

• Netflix uses AV1 for android, AVC (Advanced video coding aka h.264) for others and experimenting with HEVC (High efficiency video coding aka h.265) for UHD streaming.
• Prime video, Disney plus and Hulu services use HEVC for UHD streaming.
• YouTube mainly uses MP4 and AVC with support for a variety of codecs such as MOV, MPEG4, AVI, FLV, WMV, MPEG PS, 3GPP and WebM.

HEVC was released many years ago but it is gaining popularity recently mainly due to the difficulty of availability and difficulty of updating the hardware necessary to handle these codecs.

General architecture

Figure 1 illustrates the general architecture of any video codec. This is a high level block diagram that does not illustrate the more specific components of a video codec. However, the aim is to understand the general working of a codec. Subsequent blog posts will analyze codecs in much more detail.

Essentially, the goal of a video codec is to compress a video to a reasonable size to allow efficient storage and transmission. There are two main components of a video codec, the encoder and the decoder.

The encoder converts the raw video into a format that is suitable to store or to transmit to another location. The input video is taken in frame by frame into the prediction block. The prediction block, as the name suggests, predicts the pixel values using different techniques. For instance, AVC does block based coding that uses inter and intra frame coding. The transform block converts the data from one form to another. AVC, for example, uses the Discrete cosine transform in its transform block. Finally, the encode block converts the output from the transform block into a form that is compressed and suited to transmit.

The decoder performs all the actions in the encoder block but in reverse order. It first decodes the data to get back the output of the transform block in the encoder. The inverse transform block converts the data into a form before the transform algorithm is applied. Finally, the video is reconstructed in the reconstruction block.

Recent developments

Three new video codecs are going to be released in 2020 by Moving Pictures Expert Group (MPEG): Versatile Video Coding (VVC), Essential Video Coding (EVC) and Low Complexity Enhancement Video Coding (LCEVC).

VVC is the result of the partnership between MPEG and Video Coding Experts Group (VCEG). According to a press release from the International Telecommunication Union or ITU, VVC requires half the bit rate of HEVC to achieve the same video quality. This will allow streaming services to provide UHD content at the speed that HD services are provided today. It also introduces other evolutionary improvements such as Palette Coding and Adaptive Loop Filtering.

Preliminary performance tests show that EVC produces a 23–25% reduction in bitrate compared to HEVC and 30–35% compared to AVC. It also produces approximately 60% reduction in the encoding time compared to AVC. However, it is still under development and these are just initial results.

LCEVC tries to solve the problem of upgrading video services to higher resolutions without ignoring users with older video devices. It allows all web browsers to decode high resolution videos without any additional plugins.

That was everything needed to understand how codecs work. This post focused on video codecs, however, with the general understanding of how a video codec works, trying to understand how audio codecs or image codecs work will be much easier. With the growing demand for UHD content and the rising popularity of streaming services, video codecs like VVC, LCEVC, and EVC will allow these services to provide content as well as allow users to be entertained with limited bandwidth.

External links

List of codecs — https://en.wikipedia.org/wiki/List_of_codecs

In depth technical breakdown of VVC — https://bitmovin.com/vvc-video-codec/

Press release by ITU-T — https://www.itu.int/en/mediacentre/Pages/pr13-2020-New-Versatile-Video-coding-standard-video-compression.aspx

Official website of LCEVC — https://www.lcevc.com/

Decoding codecs. was originally published in Multimedia Things on Medium, where people are continuing the conversation by highlighting and responding to this story.