MX Content Factory: A Distributed Auto-Scaling Content Asset Processing System
Abstract
MX Player now presents millions of online content to our users. The contents are ingested from different vendors that follow diverse standards. They are of mixed quality and different formats. To ensure the best user experience, the contents need to be verified and processed before exposing on the app. For better system architecture, a consolidated content asset shall be used throughout the entire MX Player system. We have designed and developed the MX Content Factory(MCF), a distributed content assets processing system, to accomplish the goal.
The MCF system manages the whole procedure of content processing. It converts the content’s video, audio, subtitles, and thumbnails to unified formats and packages them to a structured content asset. The content asset is provided to downstream systems to support content consumption on MX Player.
It is a distributed system built on AWS EC2 cluster. It can auto-scale according to the projected workload and available resources. Thus the system is capable of handling sudden and emergency tasks at high speed and low cost.
This article briefly introduces the MCF system, including its architecture, functionalities, and components.
System Overview
Figure 1 illustrates the system position and main function. The system cooperates with the Content Management System(CMS) and Distributed Transcoding System[1] (introduced in another article on our blog) to accomplish the overall content assets processing pipeline.
The contents are ingested to CMS from a variety of vendors. CMS handles the metadata and stores source content components, such as video files, dubbed audio files, subtitle files, and thumbnail images. MCF receives source contents from CMS and processes them. The video files are sent to the distributed transcoding system to perform video transcoding. Audio files, subtitle files, and images are converted into unified formats by the MCF system. The processed results are packaged into a content asset stored in AWS S3 storage. The content asset metadata is stored back to CMS and provided to other downstream systems. The content asset in AWS S3 storage and content asset metadata in CMS is used for video streaming, downloading and other content consumption features in MX Player.
System Architecture
The MCF system architecture is presented in Figure 2. The system is a distributed system built on AWS EC2 cluster. The core component is a distributed task pipeline implemented on Celery[2].
API module is an HTTP API service built on the Django framework[3]. API module receives CMS requests containing source content components information. The Broker gets the content components data, generates content asset task and pushes the task to the task queue. We utilized RabbitMQ[4] as the task queue to support priority tasks.
The workers obtain tasks from the task queue and execute the central functionalities of the system: convert content formats and generate content assets. After finishing the task, the worker generates 3 output. Task logs are stored in a MySQL database for monitoring and auto-scale. The content asset is stored in AWS S3 for CDN. The content asset data is stored back to CMS for other systems to use. The workers also report their status by the message queue.
The auto-scale module controls the overall worker number in the system. It collects task logs from MySQL database and gathers workers’ status by the message queue. The auto-scale module controls the shutdown and startup of the workers by commands over the message queue and AWS API requests.
System Functionalities
The MCF system’s main purpose is to process ingested contents and generate content assets. There are four major parts of the main function: video processing, audio processing, subtitle processing, and thumbnail processing. For better system performance and robustness, there are also systematic components handling auto scale and monitor.
Video Processing
The MCF system cooperates with other transcoding systems. Three transcoding systems are applied for failover.
The Distributed Transcoding System is the default selection. It splits videos into small segments and transcodes them in parallel, thus can provide extremely fast video transcoding. Also, it utilized AI technology for bitrate optimization, so the output videos are smaller in size and better in quality.
The second choice is a multi-node transcoding system we built earlier. If both of them fails, the MCF system employs AWS Elemental services as a failover.
Although the MCF system does not transcode videos by itself, it implements other related work which can be divided into two phases: pre-transcode phase and post-transcode phase. The main procedures are illustrated in Figure 3.
In the pre-transcoding phase, the system samples the source video and generates a video hash used as the identifier of the source video. It extracts the source video information(resolution, bitrate, codec, etc) for video checking and generating transcode parameters. Then the system checks if the source video satisfies our specifications, such as minimum resolution, bitrates, audio tracks assignment, etc. It also checks if there are issues in the source video. Some problems can be fixed by the system. If the source video does not satisfy the specifications or has unfixable issues, the content generation task is failed.
Next, the MCF system splits the source video into video tracks, audio tracks, and subtitle tracks. Audio tracks and subtitle tracks will be processed in corresponding modules. If there are fixable issues in the video tracks, the system will repair them automatically. For example, to use Distributed Transcoding System, the video track’s I-frame assignment has to meet certain standards, due to the split method of video segments.
For better compatibility of different screen sizes of devices, the MCF system detects black borders of the video. The black borders will be cropped while transcoding. Figure 4 shows one example of black border detection. Some videos may have black frames at the start and end of the video. These black frames are also detected and removed for better user experience.
Finally, the MCF system generates transcode parameters based on the source video information, detection results, and transcode targets. These parameters are sent to the transcoding system.
After transcoding finish, the post-transcoding phase will be triggered. The MCF system merges transcoded video segments, transcoded audio tracks and subtitle tracks into separate video files of different resolutions. If the content needs to use DRM, the system utilized Widevine[5] and FairPlay[6] for video encryption. The processed videos are used as downloadable files on MX Player. And they are segmented and packed to DASH[7] and HLS[8] streams by employing Shaka Packager[9].
The MCF system assembles downloadable files, DASH streams, HLS streams, and processed thumbnails into a unified format content asset. The result content asset is uploaded to AWS S3 as CDN origin. The content asset’s metadata is stored back to CMS for downstream systems to use.
Audio Processing
As shown in the Video Processing section, the system extracts and transcodes audio tracks. The audio tracks are transcoded in different bitrates. Different bitrates audio tracks are associated with the corresponding video streams to form auto-adaptation groups in the output online streams.
For better compatibility of a variety of mobile devices, the system uses stereo channel layouts in the output audio tracks. However, the source videos may have different channel assignments. If the source video has only one mono audio track, the FC channel is copied to FL and FR channels. For layouts with more than 2 channels, the system downmixes them to FL and FR channels. Figure 5 illustrates how the MCF system converts 5.1 tracks to a stereo audio track.
For dubbed audio tracks, the system verifies if language tags are properly added to every audio track and if there is one and only one default audio track.
Subtitle Processing
To be compatible with different types of devices(Android, IOS, mobile web, desktop web), we chose WebVTT[10] as the format of subtitles. The MCF system converts the input subtitle tracks into WebVTT format. The input subtitle tracks are extracted from the source video file or provided as standalone subtitle files. Also, for dubbed subtitle tracks, the system verifies if all of them are tagged with proper languages.
Thumbnail Processing
There are four phases of thumbnail processing: Auto Generation, Preprocessing, Resizing, and Compressing.
Get Wei Zhuo’s stories in your inbox
Join Medium for free to get updates from this writer.
Auto Generation
Most of the thumbnails are provided by our vendors. But in some cases, we may need to generate thumbnails by ourselves. The MCF system employs Yahoo Hecate[11] to auto-generate thumbnails from video files. Yahoo Hecate selects attractive thumbnails by analyzing various visual quality and aesthetic metrics of video frames and performs a clustering analysis to determine the relevance to video content, thus making the resulting thumbnails more representative of the video[12]. The MCF system also utilizes the FreeType[13] engine to add title text on the generated thumbnails. We use OpenCV[14] to detect faces in the thumbnails to optimize the position of titles. Figure 6 shows one example of the auto-generated thumbnail.
Preprocessing
In this phase, the system evaluates the quality of the thumbnails and detects black borders of the thumbnails. The low-quality thumbnails(for example, low resolution, abnormal aspect ratio, etc) are rejected. Similar to the video processing, the black borders of the thumbnails are cut in the next phase to be better compatible with different device sizes.
Transcoding and Resizing
To support different types of devices, we select JPEG and PNG as the main formats of thumbnails. All input thumbnails are transcoded to JPEG or PNG with alpha channel. The black borders detected in the previous phase are cropped while transcoding.
When resizing the thumbnails, the source image’s aspect ratio may be different from the target’s. For better user experience, the system handles such gaps in three ways. If the difference is minor, the system will stretch the image a little bit. If the gap is medium, the image will be cropped first and stretched. Face recognition is also used to decide the crop position. If the aspect ratio difference is big enough, the system will add black margins to the image.
Compressing
Thumbnail size has a significant influence on the app launch speed. Smaller image size will make the app to launch faster, thus providing better user experience. The MCF system employs guetzli[15] to compress JPEG images and utilizes pngquant[16] to compress PNG images. The system also provides WebP[17] format thumbnails to clients that support this format for better compression rate.
Auto Scale
The content asset processing is resource consuming. It requires a lot of CPU, memory, network and SSD resources. We need a large cluster to process the tasks to minimize the queuing time and make our users watch these contents as soon as possible. However, the processing tasks are often come in large batches, because the content ingestion procedure is always done in batches. Maintaining a large cluster at all time is not efficient because sometimes the cluster is idle. Although the task queue is empty, every machine in the large cluster is costing money. To efficiently handle large batch tasks and save money when there is no task, we want the system to be able to upscale automatically when there are many tasks in the queue. As tasks are consumed and processed, it can automatically downscale the machines.
As shown in Figure 7, the automatic scaling system mainly consists of three parts: the broker, the worker, and the controller. They cooperate to ensure that our automatic scaling system operates smoothly. Since scaledown is more complicated, let’s describe it first.
Broker
As described in the System Architecture section, the broker is responsible for providing APIs for other programs to request. When the broker receives the task request from CMS, it will check the request parameters. If the check fails, it will return an error message. If the check passes, it will prepare to dispatch the task, the broker will check if the Redis has a lock. This lock is set by the controller. If the lock is set, waits for the controller to release the lock and then dispatches the task. If not, the broker dispatches the task directly.
Worker
The worker is a docker container running on EC2. It is responsible for the content asset processing operations. When the worker executes a task, it adds a task record in the Redis which indicates that the current worker is busy. It will update the task record when the task is finished. Both of these messages are used by the controller to determine if the worker is idle.
Controller
The third part is the controller, it checks transcoding worker status in a short interval. Before checking, it will set a lock in the Redis, if the broker receives a task at this time it will wait until the lock is released before dispatching task. The lock prevents task lost when terminating workers. Tasks may arrive at the worker which controller determines to terminate and that leads to task lost. There are two steps when terminating a worker. The first step is remotely closing the connection between the worker and the message queue, ensuring that the worker to be stopped does not receive new tasks. Because terminating AWS EC2 machine takes around 10s, if there is a task arriving during this 10s gap, the task will be lost without the first step. Then the controller sends the terminate command to stop the worker and finally releases the lock in the Redis.
The system scales the cluster up according to the queuing tasks and the resources in the current cluster. When there are too many tasks in the queue and it exceeds the processing capacity of the system, the scaleup procedure will be triggered. After upscaled the cluster, we record the current time. There is a 20 minute cooldown time between two upscaling. The reason for setting the cooldown time is that the machine needs to pre-install some libraries and download the docker image from AWS S3. Starting a worker needs around 8 minutes. Because the checking time is so short, if there is no cooldown time, the machine will continue to be applied during the startup.
This automatic scaling system provides flexibility to add or change upscaling or downscaling strategy. Now we don’t need to manually upscale or downscale the cluster, which saves time and money.
Management and Monitoring
Last but not least, the system has a management and monitor module for easier maintenance. The module is built on Celery Flower[18].
For the management function, it can control the parameters used in content asset processing, for example, how “black” is a pixel to be identified as a black border? What’s the threshold for detecting the aspect ratio gaps? It also manages a set of preset templates of parameters to help experts to fine-tune the output content assets.
For the monitor function, it monitors all the running metrics of the system, such as task queue status, task number, worker cluster status, worker number, and failed tasks, etc. It also detects systematic anomalies and sends out alerts.
Summary
MX Content Factory is a content asset processing system used in MX Player OTT business. It converts the source content’s video, audio, subtitles, and thumbnails to standard formats and assembles them into a unified content asset, which is used by downstream systems and presented on the app clients. The system utilizes diverse techniques to provide the best user experience and high compatibility of different devices. It is an auto-scale system that is capable of processing large batches of tasks in a short time and at low costs.
References
[1] Distributed Video Transcoding System, https://medium.com/mx-player/distributed-video-transcoding-system-cdfff0d42294
[2] Celery: Distributed Task Queue, http://docs.celeryproject.org/en/latest
[3] Django: The web framework for perfectionists with deadlines, https://www.djangoproject.com
[4] RabbitMQ: Messaging that just works, https://www.rabbitmq.com
[5] Widevine: Leading Content Protection for Media, https://www.widevine.com
[6] FairPlay, https://developer.apple.com/streaming/fps
[7] DASH: Dynamic Adaptive Streaming over HTTP, https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP
[8] HLS: HTTP Live Streaming, https://en.wikipedia.org/wiki/HTTP_Live_Streaming
[9] Shaka Packager, https://github.com/google/shaka-packager
[10] WebVTT: The Web Video Text Tracks Format, https://w3c.github.io/webvtt
[11] Yahoo Hecate: Automagically generate thumbnails, animated GIFs, and summaries from videos, https://github.com/yahoo/hecate
[12] To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videos, https://arxiv.org/abs/1609.01388
[13] FreeType: a freely available software library to render fonts, https://www.freetype.org
[14] OpenCV, https://opencv.org/
[15] Guetzli: Perceptual JPEG encoder, https://github.com/google/guetzli
[16] Pngquant: a command-line utility and a library for lossy compression of PNG images, https://pngquant.org
[17] WebP: A new image format for the Web, https://developers.google.com/speed/webp
[18] Celery Flower: Real-time monitor and web admin for Celery distributed task queue, https://flower.readthedocs.io/en/latest
