Distributed Video Transcoding System
Overview
Video Transcoding is the core technology used in the online video industry. The purpose of transcoding is to compress full-resolution, high-resolution, huge videos to multiple resolutions such that it can be adaptively streamed on various smart devices across network connections. For this, videos are sliced, and then each fragment is transcoded into multiple resolutions and then merged into a streaming format. All online videos we see on smart devices are served in this way.
Transcoding is necessary but comes with a high cost. Many video service companies, CDN companies, and cloud service providers offer transcoding services. For example, AWS transcoding services are expensive, but their performance is not very high and their stability is not strong. Using third-party transcoding services also greatly limits flexibility for audio track processing, subtitle processing, business pipeline, temp requirements and so on.
Transcoding costs are only a part of the total cost of providing an online video service, there are CDN costings too which are high. If your video is still very large after transcoding, it will end up incurring huge storage and bandwidth costs.
Whenever a new video releases, businesses demand that it be transcoded and made live as soon as possible. At the same time, user experience requires for a smooth and clear viewing experience. To solve for these problems, we developed a distributed transcoding system, which saved us a lot of transcoding costs, enabled us for flexible expansion and optimisation and greatly improved transcoding speeds. This system also supports high-concurrency multi-video transcoding, adaptive adjustment, and ultimate bitrate optimisation – one bitrate for one video which saves our bandwidth, saves user traffic, and truly achieves a clear and smooth streaming experience. Let me explain the system in more detail.
Video Download and Access
High Concurrency
For distributed transcoding, each segment of video is concurrently requested by multiple AWS-LAMBDA Compute Units or AWS-EC2 Compute Units. Use AWS-LAMBDA here is because it can increase to thousands of workers at anytime and no need to wait for scaling. The original video itself is stored on AWS-S3, but AWS-S3’s concurrent request capability can’t support simultaneous requests for different segments of a video, which results in high request latency and long transcoding time, and it is not acceptable in business too, meanwhile since AWS-LAMBDA is billed on time, this brings about an increase in the cost of use. So from the perspective of transcoding speed and cost savings, we need to be able to read and slice each fragment of the video in a stable and high-concurrency manner. The solution is to deploy our own file server.
Minimum Disk Usage
At the same time, in order to reduce the amount of data transmission, we use HTTP-RANGE to access the video, so that access to video fragments only needs to specify RANGE according to time. FFMPEG also supports RANGE mode to request video fragments for transcoding. In order to support high concurrency, the file server must be deployed in multiple machines in order to withstand a large number of concurrent requests. If a video is stored in multiple copies on multiple file servers, requests for different segments of a video can be answered with different file servers. But in fact, we didn’t do this, because if each file server can support concurrent requests for the same video, then this video will be downloaded and stored multiple times, which will bring extra burden to the network bandwidth and disk storage. Instead, we only store one video in a file server, and all requests for this video will arrive at the file server. Of course, the performance of this file server must be good enough. This is equivalent to replacing a large number of low-performance machines with a small number of high-performance machines to achieve stable high concurrency. The benefits are obvious, eliminating the need to repeatedly download and store video without having to address the issue of low-performance node failure. And the high-performance node fails? Will only affect the video transcoding task on this file server. We only need to re-initiate the failed task, and its video will be processed by other file servers that are alive.
Intelligent Load Balancing
If you just deploy a file server to solve the problem of stable and high concurrency, then the problem is a bit simple. We will also face the problem of request unbalanced. If a file server is providing a request for some segment of video and a request for another video is coming, the file server resource will be tight and busy. Other file servers may be idle and resources are idle. Here, we will adopt a competitive strategy, or more accurately, a polite strategy. The video download task is not dispatched by the upper layer, but by the file server for competition or courtesy. We will monitor the video transcoding queue in real time, and can monitor the number of transcoding subtasks that need to request the file server and still waiting in the queue (the fragment size of the transcoding subtask is uniform), so the load situation of file server can be determined in the future. The file server that may be heavily loaded in the future will give up the opportunity to download new videos to a file server with a small load. After a certain period of time to adjust convergence, the load of all file servers will be more balance. This is equivalent to achieving simple load balancing, but this is different from the traditional load balancing problem.
Automatic Health Checking
What if the load on the file server itself is high and even reaches a critical value? Will cause subsequent requests to avalanche. We let the file server monitor its own resource load in real time and decide whether to download the new video immediately based on the load. The monitored load includes CPU, MEMORY, NETIO, DISK, and so on. These indicators are also reported to the master monitoring system and will be used later. If the load is high, the file server will wait until the resources are sufficient. This obviously blocks, but why is it so designed? Because we need to ensure that the high-priority completion of the task which is in transcoding is stable, it cannot be overly affected by subsequent tasks. At the same time, it does a layer of protection against the file server, so that it is not crushed by too many tasks. However, in order to alleviate the problem of excessive file server pressure, we will use more file servers for a large number of transcoding tasks to expand.
Timely Disk Recycling
The disk load problem needs to be described separately. The original video files are very large, some even reach more than 200GB. In order to increase the disk read and write rate, we use high-speed SSD disks. And because the disk is expensive, we can’t use too much disk space, so we must frequently swap in and out of the video. Here we will monitor the usage of the downloaded video in real time. If the disk space is not enough to hold the next video, we will wait and try to clear the video that is no longer accessible until there is enough disk space. All of this will be automated monitoring to ensure that the transcoding system runs stably and efficiently. We will find that it is the main goal to complete the task which is in transcoding at the maximum rate, and it will get the highest priority.
Split Main Task and Trigger Sub Task
Split Main Task
Video transcoding is very CPU-intensive. That’s why we implemented this distributed video transcoding system and with the system we can finish transcoding a long video in a timely manner.
For ABR streaming, each video needs to be transcoded into different profiles with different resolutions and bitrates. For each transcoding with a specific resolution, we call it a ‘Main Task’. In order to make the transcoding highly concurrent, we split the ‘Main Task’ into many small ‘Sub Task’. Each ‘Sub Task‘ represents the transcoding task for a segment of the long video. And thus these small ‘Sub Task’ of the same or different ‘Main Task’ can then be handled concurrently and quickly on different computing units.
Trigger Sub Task
Once the main tasks are split into sub tasks, we put the sub tasks into an internal queue for next stage handling. And then the sub tasks in the queue are picked out and sent to EC2-SLAVE or AWS-LAMBDA for transcoding.
All MX’s EC2 instances can be an EC2-SLAVE for transcoding. When sending a sub task for transcoding, we check whether the existing EC2-SLAVEs are sufficient or not. If yes, the sub task will be sent to an EC2 instance for transcoding, otherwise it will be sent to AWS-LAMBDA for transcoding. In this way, we don’t need to keep a large number of EC2 instances and in the meantime, we can still make the video transcoding finished in a timely manner.
Monitor Everything
The entire distributed transcoding system needs a real-time monitoring system. At present, the main monitoring is reflected in: machine level resource monitoring, performance monitoring; Business level machine load monitoring; transcoding queue level congestion monitoring; transcoding task level status monitoring.
Machine Level Resource Monitoring
Use AWS-CLOUDWATCH to monitor transcoding cluster machine resource usage and master machine load in real time. It is auxiliary monitoring and is mostly used for historical query analysis.
Business Level Machine Load Monitoring
The file server reports its load status in real time. The upstream module will rely on this load to decide whether to trigger more transcoding units to control the pressure on the file server. The AWS-EC2 transcoding unit will report its load in real time. It will decide whether to take up more computing resources of the host machine to transcode according to the load of the host machine.
Queue Level Congestion Monitoring
In order to achieve the high concurrency capability of asynchronous, our distributed transcoding system uses message queues to control between modules. Congestion monitoring of the queue is required. If some queues are congested, the upstream distribution rate is controlled. If the queue is unbalanced, the queue should be balanced as much as possible by preemption or politeness to avoid load unbalanced for some machine at downstream.
Task Level Status Monitoring
Each transcoding task undergoes multiple state transitions from the beginning to the end of the creation, and each state is recorded and monitored. Through these task status, we can monitor the status of all transcoding main tasks and their subtasks in real time, and then observe the status of the distributed transcoding system. At the same time, we also calculate the cost of each transcoding task in real time. Transcoding task status monitoring is necessary. In addition to manually viewing and resolving bugs, you can automatically retry some failed subtasks so that the entire transcoding task does not fail because of a subtask’s timeout or failure.
Main Task Monitoring
For the transcoding main task, the following state transitions are experienced:
Get Wei Zhuo’s stories in your inbox
Join Medium for free to get updates from this writer.
Ready: The transcoding task has just been set up and is ready to start scheduling.
Downloading: The file server preempts the transcoding task and starts downloading the video in the task. It will transfer to the Failed state if download failed.
Downloaded: The file server downloads the video and prepares the transcoding compute unit to read the video fragment. It will transfer to the Failed state if download failed.
Accepted: The transcoding task is received by the executor, the status is transferred to Accepted, and the transcoding main task is split into multiple transcoding subtasks, that is, one transcode subtask for each video segment.
Running: Any transcode subtask starts running and the main task is marked as Running.
Succeeded: All transcode subtasks are successful, and the main task is marked as Succeeded.
Failed: Any transcode subtask eventually fails, and the main task is marked as Failed.
Sub Task Monitoring
For a transcode sub task, the following state transitions are experienced:
Ready: The Ready status is split after the transcode subtask is split from the main task.
CRFRunning: After being scheduled and executed by the transcoding compute unit, the video complexity based on CRF is calculated first.
CRFSucceeded, CRFFailed, CRFTimeout: The value of complexity is saved if computed successfully, and the default value is used for failure or timeout.
CRFReady (CRFFinished): Enter the CRFReady status before starting the official transcoding.
Running: The subtask starts transcoding.
Succeeded: Subtask transcoding was successful.
Failed, Timeout: The transcoding failed or timeout twice, and moved to CRFReady, waiting to continue scheduling. If the subtask eventually fails or timeout, the main task will also be marked as failed.
Low Cost Transcoding
AWS-LAMBDA
The transcoding service mentioned here refers to the part that transcodes subtasks. The main transcoding task has been split into multiple small subtasks. These small subtasks have one feature: very much, very fast. For such transcoding subtasks, each subtask can be completed with a small computational resource. However, since transcoding is a CPU-intensive calculation, CPU resources are mainly required. It is the AWS-LAMBDA service that provides a large amount of low-cost CPU computing resources, which is much cheaper than using a dedicated AWS-EC2, because AWS is also using its idle computing resources to provide services, so each compute unit of AWS-LAMBDA has only a few minutes of execution time and is charged according to time.
EC2-SLAVE
But it’s not economical to use too much AWS-LAMBDA, so we developed our own AWS-LAMBDA-like module, which we named EC2-SLAVE. It can be deployed on almost all of MX’s regular AWS-EC2 machines, maximizing the free computing resources of the machine. It monitors the computing resources of the host machine in real time. If it finds that the computing resources are idle, it calculates the number of computing units available for transcoding and then obtains the appropriate number of transcoding subtasks to execute. Since the transcoding subtask is very fast, each computing unit will only take up less CPU time and will be released immediately. At the same time, in order to avoid a perceptible impact on the host, it will retain enough computing resources to the host machine as much as possible.
Dispatching Control
If AWS-LAMBDA and EC2-SLAVE are activated too much, it will cause a large number of access requests to the file server, which will make the file server response slower. On the one hand, the delay is increased, which causes the transcoding to become slower; on the other hand, the AWS-LAMBDA execution time becomes longer due to the longer reading time, which increases the cost unnecessarily. So we must ensure that the file server always runs in a high concurrency, low latency state. When we distribute the transcoding task to AWS-LAMBDA and EC2-SLAVE, we control the distribution rate based on the machine load reported by the file server. This explains why the file server mentioned above reports its load to the monitoring system.
High scale ability
File server and EC2 instances number all can be auto scaled upon the job task queue length, it is very easily to scale large or small number. This kind of architecture design can high fully meet various business needs.
Summary
In this article, I focused on how our distributed transcoding system stabilises high-concurrency transcoding services. It also illustrates how to save costs under the premise of ensuring stable and high concurrency. The cost savings here are mainly considered from the perspective of computing resources.
In my next article, I will talk about the bitrate optimisation method, that is, how to reduce the bitrate as much as possible under the premise of guaranteeing the quality of transcoding. The reduction in bitrate is significant – on the one hand, it saves us the bandwidth cost of being an online video service provider, and on the other hand allows users to watch smooth, high-quality videos while saving their traffic costs. Stay tuned. ~
