<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bo Zhang on Medium]]></title>
        <description><![CDATA[Stories by Bo Zhang on Medium]]></description>
        <link>https://medium.com/@bozhang-26963?source=rss-c84299d4b33b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yz1E32fbB_-PPHQWyGddow.jpeg</url>
            <title>Stories by Bo Zhang on Medium</title>
            <link>https://medium.com/@bozhang-26963?source=rss-c84299d4b33b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:22:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bozhang-26963/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI-assisted, ROI-driven Video Encoding]]></title>
            <link>https://bozhang-26963.medium.com/ai-assisted-roi-driven-video-encoding-2780b12e214e?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2780b12e214e</guid>
            <category><![CDATA[video-encoding]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[yolov8]]></category>
            <category><![CDATA[ai-video]]></category>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 20:36:13 GMT</pubDate>
            <atom:updated>2026-09-07T19:16:10.621Z</atom:updated>
            <content:encoded><![CDATA[<h3>AI-assisted, ROI-based Video Encoding</h3><p>Bo Zhang, 09/06/2026</p><p>The video files in your catalog are too large. You want to compress them to save storage and CDN cost. CRF 30 was used during transcoding to deep-compress the videos. The file size shrunk but the visual quality looks horrible.</p><p>AI-assisted, Region of Interest (ROI)-driven encoding eliminates this compromise. Instead of applying uniform compression across the entire frame, ROI-driven encoding preserves high fidelity in perceptually critical areas — like faces or on-screen text — while heavily compressing background regions that viewers naturally ignore. The result is substantial bitrate savings without a noticeable drop in perceived quality.</p><p>This article presents <a href="https://github.com/maxutility2011/roi_video_encoder">roi_video_encoder</a> — my own proof of concept for an ROI-driven video encoder using <strong>YOLO26</strong> via <strong>Roboflow’s</strong> video inference engine and <strong>x264</strong>&#39;s addroi filter.</p><p><strong>TL;DR</strong></p><p>Experimental results demonstrate that <a href="https://github.com/maxutility2011/roi_video_encoder">roi_video_encoder</a><strong> achieves significant bitrate savings while maintaining high perceptual quality.</strong> Furthermore, because YOLO is a lightweight model, the pipeline processes video frames with minimal overhead, maintaining high throughput.</p><h3>The Workflow</h3><ol><li><strong>Decode:</strong> Load the video input and decode it into a sequence of RGB pictures.</li><li><strong>Detect:</strong> Process each frame through YOLO26 to identify target ROIs (e.g., persons) and extract their bounding boxes.</li><li><strong>Attach ROI Metadata:</strong> Pass the bounding box coordinates to x264 as AV_FRAME_DATA_REGIONS_OF_INTEREST frame side data prior to encoding each AVFrame.</li><li><strong>Configure Parameters:</strong> Set a higher base Constant Rate Factor (--crf) for the entire frame to save overall bitrate, then apply a negative QP (Quantization Parameter) offset (-qoffset) to preserve high visual quality specifically within the detected ROIs. This strategy intentionally sacrifices background detail to maintain high perceptual quality where viewers focus most.</li><li><strong>Encode:</strong> Encode the frames using <strong>PyAV</strong> (the Pythonic bindings for FFmpeg) configured with x264 to output the optimized video stream.</li></ol><p>While leveraging a ROI-driven bitrate control method, ROI encoder still uses x264 for encoding video pictures. No pixel in the output video is generated by any AI models. CV models are only used to assist the encoder in achieving better quality/bitrate tradeoff.</p><p>Github repo: <a href="https://github.com/maxutility2011/roi_video_encoder">https://github.com/maxutility2011/roi_video_encoder</a></p><p>The source code was entirely written by Claude code, while evaluation was done manually.</p><h3><strong>Evaluation</strong></h3><p>I encoded two public test videos — <a href="https://drive.google.com/file/d/1RN7Sw7asVozB8td6tBDBL-2341UsXg3J/view?usp=sharing">video 1</a> and <a href="https://drive.google.com/file/d/1lVESIy4Zvh4qeyzJTtRk-GMWWvcgRes1/view?usp=sharing">video 2</a> using both the ROI encoder and x264 (but with a whole-picture CRF target), then compare the two encoders’ outputs in terms of visual quality, bitrate efficiency and encoding speed.</p><p><strong>Test videos</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2fx8NbPjPFxf_PJt1Qui_A.png" /><figcaption>Table 1: Test videos from Pexels.com</figcaption></figure><p><strong>Test server — GCP n2-standard-4 (4 vCPUs, 16 GB Memory)</strong></p><p>GPU was not used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*GypFTRhRnzV3cXR-XTY-3A.png" /><figcaption>Table 2: Encoding server</figcaption></figure><p><strong>Evaluation setup</strong></p><p>The ROI encoder provides the following configuration parameters:</p><ul><li>“ — roi [roi_type1],[roi_type2],…”: the type of ROIs (i.e., objects to detect), e.g., “person”, “text”.</li><li>“( <em>— conf 0.4</em>)”: As long as the YOLO26 model detects a object that matches the target ROI type (e.g., person) with 40+% confidence, the detected region is considered as a ROI region.</li><li>“ — crf”: The base (background) CRF for the whole picture, e.g., 37.</li><li>“ — qoffset”: QP reduction, e.g., -0.28.</li></ul><p><strong>Video 1 results</strong></p><p><strong>roi_encoder</strong> command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*BnnedYNV0HvPLmBjK7dtHw.png" /></figure><p>Specifically, this command specifies the detection of persons in the video via “ <em>— roi person</em>” option. As long as the YOLO26 model detects a person with 40+% confidence ( <em>— conf 0.4</em>), the detected region is considered as a ROI. The command also instructs the encoder to use CRF 37 (low quality) for encoding the whole picture by default, and uses a QP reduction of 28% ( <em>— qoffset=-0.28</em>) to enhance the visual quality for the ROI regions.</p><p><strong>x264</strong> command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*O4zYVgVjlFmraI1nXVFMfg.png" /></figure><p>The x264 command uses a target CRF value of <strong>27</strong> as compared to <strong>37</strong>. This configuration aims for a significantly higher quality at the cost of much higher output bitrate. However, the results shows the opposite — the ROI encode shows a higher visual quality and at same time with a 16.4% lower bitrate, beating x264 in both quality and bitrate efficiency.</p><p>Figure 1 shows the x264 output (left) side by side with roi_encoder output (right). The x264 encode has a video bitrate of 1243 kbps while the ROI encode only has 1068 kbps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcRgs5QpJC_Tc6JKPTI0XA.png" /><figcaption>Figure 1: libx264 (left) vs. roi_encoder (right) — whole picture</figcaption></figure><p>Despite with 16.4% lower bitrate, the ROI encode shows obviously higher visual quality in the ROI regions than the x264 encode — the person’s body, as seen from fig 2–3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*R-uniNW4dYMTZyFVfZhNAg.jpeg" /><figcaption>Figure 2: x264 (left) vs. roi_encoder (right) — head</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*gPpFYqBlbXqJGvet_sR2aQ.jpeg" /><figcaption>Figure 3: x264 (left) vs. roi_encoder (right) — hand</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*qmfUI9hkbpwpVQ_qWuuWGg.jpeg" /><figcaption>Figure 4: x264 (left) vs. roi_encoder (right) — cooking pan</figcaption></figure><p>In figure 2 and 3, the hair, eye and hand are apparently more clear in the ROI encode than in the x264 encode, because they received more bits. However, for the cooking pan area, x264 shows better result because this area is outside the ROI hence was encoded with the base CRF (37).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*88AoVMzZUKsDB0WJS0DnXw.png" /><figcaption>Table 3: Coding efficiency and speed — video 1</figcaption></figure><p>ROI encoder processes video 1 at a speed of 9.2 fps, not bad for a 4 vCPU, 16 GB memory, CPU-only VM.</p><p><strong>Video 2 results</strong></p><p><strong>roi_encoder</strong> command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*HFs5We1V46qKAtACGRecJw.png" /></figure><p><strong>x264</strong> command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*ST6QADYxIs41bEmcNc6k5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qq8-uQwpjPqAjtBhNjOZ-Q.png" /><figcaption>Table 4: Coding efficiency and speed — video 2</figcaption></figure><p>For test video 2, the x264 encode and the ROI encode show similar visual quality, as seen in figure 5–7. However, the ROI encode bitrate (1071 kbps) is 15.05% lower than the x264 encode (1261 kbps), reflecting a higher bitrate efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a2FD9cynWtD2TkZqfSpzZA.jpeg" /><figcaption>Figure 5: x264 (left) vs. roi_encoder (right) — whole picture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vnTeM9eBjKrm8RPf0osqvA.jpeg" /><figcaption>Figure 6: x264 (left) vs. roi_encoder (right) — head</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Qg6MejkpnvlbUf59JW9EEQ.jpeg" /><figcaption>Figure 7: x264 (left) vs. roi_encoder (right) — shirt</figcaption></figure><p><strong>ROI encoding time and speed analysis</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6OHl7g7O8KtigmI5Xr1obQ.png" /><figcaption>Tale 5: Time and speed analysis</figcaption></figure><p>Video 1 took a total of 75.26s to be ROI-encoded. Specifically, 4.28s was spent on decoding the input video. 50.01s was spent on detecting the ROI — person. 20.96s was spent on encoding the output. ROI detection time was 66.44% of the total encoding time. (YOLO) model loading time was excluded.</p><p>Video 2 took a total of 23.84s to be ROI-encoded. 1.08s was spent on decoding the input video. 16.36s was spent on detecting the ROI — person. 6.4s was spent on encoding the output. ROI detection time was 68.62% of the total encoding time.</p><p><strong>Future work</strong></p><p>In my current evaluation, I have only verified the efficacy of the method on a very small subset of test files. A tremendous amount of research and engineering work is still needed to generalize the approach to a broader and more representative test set.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2780b12e214e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting objects in live videos and streaming live annotated videos with HLS in ezLiveStreaming]]></title>
            <link>https://bozhang-26963.medium.com/detecting-objects-in-live-videos-and-streaming-live-annotated-videos-with-hls-in-ezlivestreaming-fd3017d4b14b?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd3017d4b14b</guid>
            <category><![CDATA[video-transcoding]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[live-streaming-video]]></category>
            <category><![CDATA[yolov8]]></category>
            <category><![CDATA[http-live-streaming]]></category>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 01:49:13 GMT</pubDate>
            <atom:updated>2025-03-14T13:38:55.199Z</atom:updated>
            <content:encoded><![CDATA[<h3>Detecting objects in live videos and streaming live annotated videos via HLS</h3><p><a href="https://github.com/maxutility2011/ezLiveStreaming/tree/main"><strong>ezLiveStreaming</strong></a> leverages an AI model <a href="https://yolov8.com/"><strong>Yolov8</strong></a> to enable real-time detection/identification of objects in live videos and then live streaming the annotated videos using HLS to users. Users can enable object detection in their live job requests and configure annotated video outputs in HLS streaming format. Though many existing video-based applications already support video object detection, to my knowledge, <strong>ezLiveStreaming is the first open-source live transcoding service to offer this feature</strong>.</p><p>With object detection, ezLiveStreaming enables a wide range of real-time RoI (Region of Interests) identification applications, such as match score and jersey number recognition in sports, object counting, crowd monitoring, obstacle detection, traffic sign recognition and intruder detection.</p><p>Want to try out this new feature? Check out <a href="https://github.com/maxutility2011/ezLiveStreaming/blob/main/doc/yolo.md">this page</a> to learn more!</p><h3>Demo screenshots</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EXb339BOvhGEYtgF.png" /></figure><p>Live streaming of AI-annotated video using Hls.js</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FdehQUCcFxoi8oa_.png" /></figure><p>Live streaming of AI-annotated video using Hls.js</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fJoC5TMw3XjhsvYQ.png" /></figure><p>Live streaming of AI-annotated video using Hls.js</p><h3>Demo video</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLEJ_3IrN_8s%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLEJ_3IrN_8s&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLEJ_3IrN_8s%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/db87b5c3e5e55e439880728097fa08d9/href">https://medium.com/media/db87b5c3e5e55e439880728097fa08d9/href</a></iframe><p>Live streaming of AI-annotated video using Hls.js</p><p>To explore other live streaming features in ezLiveStreaming, such as <strong>AV1 based live transcoding</strong>, <strong>DRM</strong>, start from t<a href="https://github.com/maxutility2011/ezLiveStreaming/tree/main">his document</a>.</p><p>Have questions? Feel free to email me at maxutility2011@gmail.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd3017d4b14b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Video Super Resolution for Live Video Upscaling]]></title>
            <link>https://bozhang-26963.medium.com/real-time-video-super-resolution-for-live-video-upscaling-078857e5ad60?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/078857e5ad60</guid>
            <category><![CDATA[transcoding]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[videos]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Mon, 23 Sep 2024 19:02:09 GMT</pubDate>
            <atom:updated>2025-12-20T02:17:58.124Z</atom:updated>
            <content:encoded><![CDATA[<h3>Real-Time Video Super Resolution for Live Video Enhancement: A Proof of Concept Implementation</h3><p>Recently, I have been working on a side project on Video Super Resolution (VSR) using a GAN-based machine learning model. I want to explore if VSR can be done at real-time speed. Specifically, I have built a Proof of Concept (PoC) VSR upscaling system which breaks an input video stream into a sequence of images, run a pre-built Pytorch image super resolution model on the images and upscale the images by a factor of 4 to enhance their visual quality, and then uses FFmpeg to combine and re-encode the images to a single high resolution output video. I also made a series of optimizations to make the whole upscaling (enhancement) process as fast as possible on my Nvidia GeForce 4060 GPU with 8GB built-in memory. My test results show that when input resolution was 108x192 (vertical) and output resolution was 432x768 (i.e., x4 upscaling), real-time VSR was possible and the overall upscaling speed was 1.1x of the input video frame rate. When the input resolution was 180x320 and output resolution was 720x1280, real-time upscaling was not achieved and the upscaling speed of my PoC system was only 0.44x the input frame rate. For both inputs, my VSR upscaling implementation achieved a significantly better visual quality than the traditional LANCZOS upscaling, and at the same time the output video bitrate was much lower than LANCZOS too.</p><p>First of all, let me explain why real-time VSR enhancement matters. For one thing, traditional signal processing based video upscaling algorithms such as BICUBIC and LANCZOS do not perform very well when the live input video uses a very low resolution, such as 320x180. Machine learning based super resolution upscaling could significantly outperform traditional technologies by enhancing the visual quality while saving bitrate. However, the biggest challenge of running VSR-based upscaling at real-time speed is how fast we can run ML inference as such operation is highly complex and time consuming even on GPU. The speed of generating ML-inferred high resolution video frames must be at least as high as the speed at which live video frames are ingested and streamed out.</p><p>A second question arises as to why we need such low live input resolutions such as 320x180? The answer is that, in many live streaming scenarios, the streamers may be located outdoor with bad cellular connections such as me doing live broadcasting of my daughter’s weekend soccer games. The soccer field is located in a quite rural area and the bad network connection simply cannot support a reasonably good live contribution quality. But I don’t want my viewers to watch poor quality videos, so what I can do is to upscale the low quality input to a higher resolution with a much better quality using super resolution. Adaptive BitRate (ABR) transcoding can then be applied on the upscaled contribution stream before delivering the video to the viewers.</p><p>GitHub link: https://github.com/maxutility2011/vsr_real_esrgan</p><h3>Implementation overview</h3><p>There are many existing machine learning models for video super resolution, such as EDVR, BasicVSR, ESRGAN, COMISR, Stable Diffusion x4 upscaler, etc. A quite comprehensive list of the models can be found in [1]. I evaluated a set of these models in terms of upscaled video quality, then picked one GAN-based model for my implementation which outputs noticeably better visual quality than most others. I also made a series of optimizations to the PoC implementation, such as serializing the PyTorch model into a TensorRT engine, etc. The PoC was written in Python and shell script, and libraries such as Numpy and OpenCV were used to process the image tensors.</p><h3>Experiment setup</h3><p><strong>Test video</strong>:</p><ul><li>Resolution: 108 x 192 (vertical) and 180 x 320 (vertical). Vertical video inputs were used because many amateur outdoor streamers simply use their mobile phones for streaming. Using vertical videos is not an speed optimization.</li><li>Bitrate: 67kbps and 149kbps (for 108 x 192 and 180 x 320 input, respectively). These are very friendly bitrate values for live broadcast contribution even with poor cellular connections.</li><li>Video duration (sec)/Frame rate (fps)/Frame count: 39.66/15/598</li><li>Color format: RGB</li><li>Input tensor shape (excluding inference batch size): (3, 108, 192) or (3, 180, 320). Given RGB color format, the number of color channels is 3. The two other dimensions are the video width and height.</li><li>Output tensor shape: (3, 432, 768) or (3, 720, 1280) as the input resolutions are upscaled by a factor of 4.</li></ul><p><strong>Hardware</strong></p><ul><li>CPU: AMD Ryzen 7 7700 8-Core Processor 3.80 GHz with 16GB RAM</li><li>GPU: NVIDIA GeForce 4060 with 8GB memory</li></ul><p>GPU was used for VSR inference only. CPU was used for running all other steps specified as follows.</p><p><strong>Upscaling steps</strong></p><p>The PoC uses the following steps for upscaling the input video,</p><ul><li>Breaking the source video into a sequence of 598 images,</li></ul><pre>ffmpeg -i $1 -vf fps=$2 images/image_%4d.png</pre><ul><li>Loading the 598 images, and pre-processing them before starting VSR inference (e.g., normaling and transposing the input tensors before feeding them to the neural network),</li><li>Copying the input tensors (processed 108x192 or 180x320 images) to the GPU and running x4 upscaling inference on the input images (upscaling by a factor of 4),</li><li>Performing post-processing on the output tensors (i.e., the upscaled 432x768 or 720x1280 images) which is basically the reverse of the pre-processing step.</li><li>Performing minimal compression on the output tensors and save them as BMP images.</li><li>Using FFmpeg and libx264 to combine and re-encode the 598 output images to a single output video.</li></ul><p>For 108x192 input and 180x320 input, inference batch sizes of 6 and 2 were used, respectively. The GAN-based super resolution model is kind of large with dozens of millions of parameters. Running such a large model consumes a lot of memory during inference which leaves only a limited amount of memory for holding the input tensors as well as the output tensors. Batch size of 6 and 2 are the highest possible values allowed by my GPU which has only 8GB built-in memory. A larger batch size could allow further acceleration (hopefully, ~10–20%), but not available on my server.</p><p>For LANCZOS-based upscaling, a upscaling factor of 4 was also used and the following upscaling command was used,</p><pre>ffmpeg -i go_fishing_180x320.mp4 -vf scale=720:1280:flags=lanczos -c:v libx264 -crf 15 -preset slower -y output_lanczos.mp4</pre><h3>Results</h3><p>Fig. 1–4 compare the source, the LANCZOS-upscaled and VSR-upscaled images for both 108x192 and 180x320 inputs in two different scenes. As can be seen, the visual quality of super resolution upscaled images are much better than the sources and LANCZOS-upscaled ones (Please view the difference on a larger screen. The difference may not be as obvious when viewed on a mobile phone). Next, let’s look at the upscaling speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3E6Osd2RQLw3e3pO4cqstA.png" /><figcaption>Fig. 1: Output images (source vs. LANCZOS vs. super resolution) — 108x192 input, scene 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xy1ZEcVoZYPearI33ukqw.png" /><figcaption>Fig. 2: Output images (source vs. LANCZOS vs. super resolution) — 108x192 input, scene 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jfn7RUPNu2iNShGC0qbXiA.png" /><figcaption>Fig. 3: Output images (source vs. LANCZOS vs. super resolution) — 180x320 input, scene 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cSPj9T5D8gHiNcPS4ADrpQ.png" /><figcaption>Fig. 4: Output images (source vs. LANCZOS vs. super resolution) — 180x320 input, scene 2</figcaption></figure><p>Table 1 and 2 show the upscaling performance of LANCZOS vs. VSR upscaling. When input resolution is 108x192, VSR was able to achieve real-time upscaling because its overall processing (pre- and post- processing, VSR inference, output image compression and saving) speed was 16.55 fps, higher than 15 fps which is the input frame rate. This represents a relative speed of 1.1x.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*Yl6xF8p0X3QLiEP0KgwGEQ.png" /><figcaption>Table 1: Upscaling performance: LANCZOS vs. Super resolution (108x192)</figcaption></figure><p>Unfortunately, when input resolution is 180x320, VSR was unable to achieve real-time upscaling because its overall processing speed was only 6.62 fps, much lower than the input frame rate, 15 fps. This represents a relative speed of 0.44x. However, this does not conclude the impossibility of real-time VSR upscaling. I think further acceleration can surely be achieved by using higher end GPU models with higher computing power, larger built-in memory and more advanced inference framework, etc. Or, I can also install another set of GPU and run two inference tasks in parallel. My GeForce 4060 is more like a gaming graphic card, not the best model for AI inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*gwG68HHwP4DU-flc173DQA.png" /><figcaption>Table 2: Upscaling performance: LANCZOS vs. Super resolution (180x320)</figcaption></figure><p>Finally, let us look at the breakdown of the average total processing time for one single video frame (Table 3). As can be seen, For 108x192 input, the total processing time per frame is 46.9 ms including 32.2 ms for VSR inference only, 4.17 ms for pre-processing the input tensors and post-processing the output tensors, 10.5 ms for compressing and saving the output images, plus some insignificant video re-encoding time.</p><p>For 180x320 input, the total processing time per frame is 131.3 ms including 89.5 ms for VSR inference only, 11.64 ms for pre-processing the input tensors and post-processing the output tensors, 30 ms for compressing and saving the output images, plus some insignificant video re-encoding time.</p><p>For both input resolutions, inference only took 68% of the total processing time. 32% of time was spent on pre-processing, post-processing, output compression and saving and video re-encoding, combined. I believe the 32% non-inference time can be further cut down to accelerate the whole process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*OUFF_44ya-NKb_sbD9D44w.png" /><figcaption>Table 3: Breakdown of total processing time per frame</figcaption></figure><p>If you have any questions about this blog, please reach out to me at maxutility2011@gmail.com.</p><h3>References</h3><p>[1] Nabajeet Barman, Yuriy Reznik, Maria Martini, On the Performance of Video Super Resolution Algorithms for HTTP-based Adaptive Streaming Applications, <a href="https://eprints.kingston.ac.uk/id/eprint/54894/">https://eprints.kingston.ac.uk/id/eprint/54894/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=078857e5ad60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DynaMediaPackager — A Library for Dynamic Media Packaging based on Shaka Packager]]></title>
            <link>https://bozhang-26963.medium.com/dynamediapackager-a-library-for-dynamic-media-packaging-based-on-shaka-packager-8945adf34e41?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8945adf34e41</guid>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Sun, 20 Mar 2022 20:09:46 GMT</pubDate>
            <atom:updated>2022-03-22T02:08:13.633Z</atom:updated>
            <content:encoded><![CDATA[<h3>DynaMediaPackager — A Library for Dynamic Media Packaging based on Shaka Packager</h3><p>A few months ago, I wrote a sample application for transmuxing video streams using the Shaka packager, <a href="https://tinyurl.com/mr8pzcr5">https://tinyurl.com/mr8pzcr5</a>. This application uses a static <strong>PUSH streaming workflow</strong> in which the transcoder/packager receives the input media stream from a local file or a network stream, processes the stream, then writes (pushes) the transmuxed segments/playlists into static files. While the PUSH model is quite straightforward, the static packager has to output multiple static copies of the same original stream to serve different types of streaming clients. This isn’t quite efficient for media storage and delivery. Therefore, the dynamic PULL workflow has becoming more popular recently in which the transcoder only outputs one single copy of mezzanine stream, then it is the job of the dynamic packager to package the mezzanine stream into multiple output streams in a just-in-time manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wx_klXe_IZJ9Pki7YqcH4Q.png" /><figcaption>Figure 1: PUSH vs. PULL streaming workflow</figcaption></figure><p>A common design of dynamic packaging is to integrate the media packager as a module into the origin streaming server. For example, we can integrate a packaging library as a module into the well-known NGINX web server. Kaltura’s NGINX-VOD-MODULE (https://github.com/kaltura/nginx-vod-module) is an example of such design. When NGINX receives streaming requests, it downloads the mezzanine stream data, then invokes the packaging pipeline as function calls to process the mezzanine data. When the packaging jobs complete in the backend, NGINX receives the processed (output) stream from a callback function invoked when the packaging pipeline finishes. In this article, I present <strong>dynaMediaPackager </strong>(<a href="https://tinyurl.com/4n57f3j3">https://tinyurl.com/4n57f3j3</a>), a single-function library for building a dynamic packager. To help people better understand its use, I developed a simple dynamic packager application (written in Golang and CGO) using dynaMediaPackager. This Go application is essentially a media streaming proxy that handles streaming requests from video players, dynamically generates and serves media segments from a mezzanine stream. I also designed a simple demo which shows how this dynamic packager can serve an HLS-ts stream out of an HLS-fmp4 original stream. While NGINX-VOD-MODULE also implements a dynamic packager, it only works for VoD stream. DynaMediaPackager works for both live and VoD, since it deals with individual media segments no matter it belongs to a VoD or live stream.</p><h3>Design of dynaMediaPackager</h3><p>dynaMediaPackager includes only a single C/C++ function called “processMediaData(…)” as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hBeAfJ8H8aJKtow-mR35Ag.png" /><figcaption>Figure 2: processMediaData function definition</figcaption></figure><p>The function takes six inputs,</p><p>- <em>char *inputFileId</em>: this is the name of the input segment. It can be anything. For example, it can be the URL of the segment that is being requested by a streaming client, e.g. “video_1.m4v” or “video_1647701005.m4v”.</p><p>- <em>char *pBufferData</em>: this points to the memory buffer storing the input segment. The buffer can store a whole fmp4 or ts segment.</p><p>- <em>size_t bufferSize</em>: this gives the size of the media buffer (i.e. the size of the input segment).</p><p>- <em>char *outputFormat</em>: this is the target output container format. Currently, only “.ts” and “.mp4” are supported.</p><p>- <em>packagingParams_t params</em>: this structure allows the client application to specify how the output segment shall be packaged.</p><p>- <em>uint8_t **outputBufferData</em>: this allows the packager to return the memory buffer storing the output segment to the client application. The output buffer is allocated by the packager, it is the client application’s job to deallocate.</p><p>- <em>size_t *outputBufferSize</em>: this returns of the size of the output buffer.</p><p>Under the hood, dynaMediaPackager uses the Shaka packager as the media transmuxing and encryption pipeline. When function <em>processMediaData</em> is called, a Shaka packager pipeline is instantiated and initialized with the packaging parameters, <em>packagingParams_t params</em>. Shaka pipeline accepts input media data using a number of methods, such as reading from a file, a network stream (e.g. udp, http) or a <strong>memory file</strong>. When the client application has the input segment buffer ready, it is passed as a function argument to dynaMediaPackager. From dynaMediaPackager to Shaka pipeline, the input buffer is passed in through a shared memory file, “<em>memory://[inputFileId]</em>”. Basically, dynaMediaPackager writes the input segment buffer into the shared memory, then Shaka pipeline reads the input buffer from the shared memory. While Shaka also supports other input methods such as reading from a local file, reading from a network stream (e.g. udp, http), shared memory file seems to be the only workaround to pass memory buffers directly into Shaka pipeline. To receive the processed segment buffer, dynaMediaPackager registers a callback function to Shaka pipeline, the buffer is then returned to the client application when everything finishes.</p><h3>Sample dynamic packager application</h3><p>A sample dynamic media packager (packager_proxy.go) based on dynaMediaPackager is provided. Packager_proxy.go implements a streaming web proxy that handles playlist and media segment requests from HLS clients. It fetches playlist and fmp4 segments from a remote location, calls dynaMediaPackager to dynamically transmux the fmp4 segments to ts segments, then sends the transmuxed segments back to the requesting HLS clients. In this way, this sample packager application can serve two types of HLS clients (HLS-fmp4 clients and HLS-ts only clients) from the same copy of fmp4 segments, hence reduces media storage and delivery cost by half. If otherwise a static packager is used, two copies of the same original stream need to be generated and delivered, one for the HLS-fmp4 client, and another one for the HLS-ts client.</p><h3>Demo</h3><p>A streaming demo using packager_proxy.go is implemented and made available from an AWS EC2 instance. An HLS-fmp4 VoD stream (HLS playlist + fmp4 segments) is hosted on the EC2 instance by NGINX on port 80. I manually made a copy of the original fmp4 playlist (stream_0.m3u8), modified it to present ts segments instead, and named it stream_0_ts.m3u8. In a production implementation, you will need a playlist converter to modify the playlist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMxX9pZDKmzhTLivYyiiDw.png" /><figcaption>Figure 3 (a): Excerpt of stream_0.m3u8</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7UmHVUQ7o28f44wQgEeq8A.png" /><figcaption>Figure 3 (b): Excerpt of stream_o_ts.m3u8</figcaption></figure><p>Then, I ran packager_proxy.go on port 8080 to serve the clients. Next, I started two HLS clients in Safari (in two different tabs), one requesting stream_0.m3u8 (the fmp4 stream), and the other one requesting stream_0_ts.m3u8 (the ts stream). Packager_proxy.go downloads the original fmp4 segments for both clients. However, for the HLS-ts client, the fmp4 segments are downloaded and dynamically transmuxed before serving, while for the HLS-fmp4 client, the fmp4 segments are passed-through directly. Because packager_proxy.go supports dynamic packaging, both HLS clients are able to play successfully, even though only an HLS-fmp4 stream is available on the remote server. The entire demo setup lives in a docker container, and is available to download at <a href="https://drive.google.com/file/d/1DYvV49oBtVaOrPoDN0aqvLag5hHWg8uH/view?usp=sharing">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yv0dTrbtkh2FDms05PaTgA.png" /><figcaption>Figure 4: Demo design</figcaption></figure><p>You need two machines to run this demo, a server machine and a client machine. First, on your server machine, run “docker import” to import the image. Run</p><p><em>docker container run -it -p 443:443 -p 80:80 -p 8080:8080 — entrypoint=/bin/bash [image_id]</em></p><p>to start the container. From inside the container, start nginx by running “<em>nginx</em>”. Go to “<em>/root/code/dynaMediaPackager/</em>”, run make to build. Then, start packager_proxy by running “<em>./packager_proxy</em>”.</p><p>Next, on your client machine, start Safari browser and open two tabs. Load the HLS-fmp4 stream which is available at <a href="http://[server_hostname]:8080/media/stream_0.m3u8">http://[server_hostname]:8080/media/stream_0.m3u8</a>. Then, load the HLS-ts stream which is available at <a href="http://[server_hostname]:8080/media/stream_0_ts.m3u8">http://[server_hostname]:8080/media/stream_0_ts.m3u8</a>. Both streams should play!</p><p>To try my AWS EC2 hosted demo, please use the following two links (remain valid through April 1st),</p><p><a href="http://ec2-52-205-164-229.compute-1.amazonaws.com:8080/media/stream_0_ts.m3u8">http://ec2-52-205-164-229.compute-1.amazonaws.com:8080/media/stream_0_ts.m3u8</a> (HLS-ts)</p><p><a href="http://ec2-52-205-164-229.compute-1.amazonaws.com:8080/media/stream_0_ts.m3u8">http://ec2-52-205-164-229.compute-1.amazonaws.com:8080/media/stream_0_ts.m3u8</a> (HLS-fmp4)</p><p>If you have any questions, please reach out to me at maxutility2011@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8945adf34e41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A sample application for transmuxing and encrypting video streams using Shaka packager SDK]]></title>
            <link>https://bozhang-26963.medium.com/a-sample-shaka-packager-based-application-for-transmuxing-and-encrypting-video-streams-5076154c074f?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5076154c074f</guid>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Mon, 06 Sep 2021 20:52:25 GMT</pubDate>
            <atom:updated>2021-09-08T19:40:39.928Z</atom:updated>
            <content:encoded><![CDATA[<p>Shaka packager is a powerful tool for performing encoder-side jobs such as repackaging and encrypting live/VoD contents, inserting ads into video streams, and streaming live videos at low-latency. Previously, I had only used its default application, “packager” for experimenting new features and building demos. I have always been looking for code samples showing how to use the Shaka packager SDK to build my own applications, but I haven’t found much. In this Labor Day weekend, I spent some time writing a very simple application which uses Shaka packager as the underlying media library for packaging and encrypting self-initialized video file. The source code of my app can be found on GitHub, https://github.com/maxutility2011/sample_app_using_shaka_packager. This repo is only a simple demo program, it is not a sophisticated video packager.</p><p>The program is very simple. First, we need to include the necessary header files from the Shaka packager repository,</p><h4>#include “packager/packager.h”</h4><h4>#include “packager/app/stream_descriptor.h”</h4><p>Next, we need to specify all the parameters (stream descriptor) needed for initializing the video pipeline, including the media input, output (but do not specify output when the output container format is a Transport Stream), segment_template (TS output only) and which tracks you wish to package into the output,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b1UfN18gChD9tMs_r9bVUg.png" /><figcaption>Figure 1: Configuring video pipeline</figcaption></figure><p>Since I’d like to encrypt the output, I also specify the parameters (encryption_params) for DRM encryption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZoJWNVOVovxwWFRne2Vpg.png" /><figcaption>Figure 2: Configuring DRM encryption</figcaption></figure><p>This includes the <strong>key provider</strong>, <strong>PSSH</strong> (Protection System Specific data) box (not needed for TS output), <strong>key id</strong>, <strong>key string</strong> and <strong>Initialization Vector</strong> — IV (<a href="https://mpeg.chiariglione.org/standards/mpeg-b/common-encryption-iso-base-media-file-format-files">ISO/IEC 23001–7</a>). For simplicity, I chose to use <strong>raw key</strong> scheme which I will provide my own clear key info directly to the encryptor. The key id, key string and IV are hardcoded in the program. A pre-generated PSSH box is also specified and will be inserted by Shaka SDK to the initialization section of the output file. Given this parameter setting, when the program calls the function “packager.Initialize(…)”, the Shaka SDK will create a video pipeline including three components, a <strong>demuxer handler</strong> for demultiplexing the input (<a href="https://github.com/google/shaka-packager/blob/master/packager/media/demuxer/demuxer.cc">shaka-packager/packager/media/demuxer.cc</a>), a <strong>muxer handler</strong> for multiplexing/packaging the output (<a href="https://github.com/google/shaka-packager/blob/master/packager/media/base/muxer.cc">shaka-packager/packager/base/muxer.cc</a>). Since we also specify DRM key provider, Shaka SDK will also create a encryption handler (<a href="https://github.com/google/shaka-packager/blob/master/packager/media/crypto/encryption_handler.cc">shaka-packager/packager/crypto/encryption_handler.cc</a>), and connect the output of the muxer handler to the input of the encryption handler. Finally, we call “packager.Run()” to start the pipeline to run the job.</p><p>The build instruction and all the dependencies can be found in the GitHub repository. After the program builds, run it like “transmux_encrypt [input_file] [output_file]”. I have tested mp4 input, and mp4/ts output. But any formats supported by Shaka SDK should also work. Shaka SDK will guess the file format by looking at the file extension or maybe the file header (or maybe not), you don’t need to specify file format. The input media file I’m using is a 4 seconds long self-initialized media segment. When we specify a TS output, my program will add a segment_template “video_$Number$.ts” to the stream descriptor, so the output file will look like “video_1.ts”. The TS segment will be encrypted by following the <a href="https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/HLS_Sample_Encryption/Intro/Intro.html#//apple_ref/doc/uid/TP40012862-CH5-SW1">FairPlay Streaming (FPS) encryption specification</a>. If we want to use the program to generate m3u8 manifests with all the DRM info, you can add “hls_playlist_name” to the stream descriptor and specify your playlist file name. By doing so, the m3u8 output will contain the encryption key (or license url), IV and encryption method. However, this parameter does not currently exist in my program.</p><p>When we specify a mp4 output, my program will assign the command-line option [output_file] to “output” of the stream descriptor. The program will output a video-only mp4 file that is encrypted using <strong>CENC/AES128-Counter mode</strong>. The output file is also self-initialized which includes a <strong>MOOV</strong> (Movie) box, a <strong>SIDX</strong> (Segment Index) box, a <strong>MOOF</strong> (Movie Fragment) box, and a <strong>MDAT</strong> (Media DATa) box (<a href="https://mpeg.chiariglione.org/standards/mpeg-4/iso-base-media-file-format/text-isoiec-14496-12-5th-edition">ISO/IEC 14496–12</a>). If you want only the initialization data or the segment media data (e.g., in DASH and HLS-fmp4 streaming, init and media segments are separate objects), you can use some tools or write your own program to split and save the two parts separately. For example, you can copy the FTYP and MOOV box (Figure 3 below) out, combine them and save as a separate initialization segment. In Figure 3, the file offset (in bytes) of MOOV is 40, and the box size is 1116 bytes, so that you know where to find the boxes and how many bytes you need to copy. You can also copy and save the rest of the segment including SIDX, MOOF and MDAT as a separate media segment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*6aH0G7E1M1nhfldAxK9j4g.png" /><figcaption>Figure 3: Structure of the output media file (MOOV)</figcaption></figure><p>Also, all the sample descriptions related to DRM encryption are inserted by Shaka SDK into the MOOF box of the output file, including SENC (Sample ENCryption), SAIO (Sample Auxiliary Information Offset) and SAIZ (Sample Auxiliary Information Size) boxes (Figure 4 below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*8GEi6JYurfzzTnISGBfwmw.png" /><figcaption>Figure 4: DRM-related boxes in the output media file (SAIZ, SAIO and SENC)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*Qx6ty-nqrrSr9tQYQgboKA.png" /><figcaption>Figure 5: DRM-related boxes in the output media file (TENC)</figcaption></figure><p>A TENC (Track ENCryption) box (Figure 5 above) is also inserted to the initialization section of the output file. One more thing to note is that, the key id, key string and IV are all provided as hex strings, so they need to be converted to byte arrays before passing into Shaka SDK. That is done by the <strong>HexToBytes</strong>() function (Figure 6 below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9GLf5JnwdmpTcZfkRbSFSg.png" /><figcaption>Figurw 6: Converting hex strings to byte arrays</figcaption></figure><p>To verify the correctness of the output, I use the Shaka default packager app to encrypt the same input file using the same encryption parameters and the following command (<a href="https://google.github.io/shaka-packager/html/tutorials/raw_key.html">https://google.github.io/shaka-packager/html/tutorials/raw_key.html</a>),</p><p><strong>./packager in=input.mp4,stream=video,output=out.mp4,drm_label=SD — enable_raw_key_encryption — enable_raw_key_decryption — keys label=SD:key_id=abba271e8bcf552bbd2e86a434a9a5d9:key=69eaa802a6763af979e8d1940fb88392:iv=73fbe3277bdf0bfc5217125bde4ca589 — clear_lead=0</strong></p><p>I found that the two outputs (one generated by Shaka default app, and one by my app) do match which is good. If your input video file is less than 6 seconds long, you need to add “ — clear_lead 0” to your shaka packager command. Otherwise, you will be surprised to find out your output file remains unencrypted. This is because by default, the Shaka packager leaves the first 6 seconds of the video unencrypted. Adding “ — clear_lead 0” will leave no clear (unencrypted) leading seconds, and Shaka packager will encrypt the entire file.</p><p>I also use an online AES-128 encryptor (<a href="https://cryptii.com/pipes/aes-encryption">https://cryptii.com/pipes/aes-encryption</a>) to encrypt a small piece of the input file (obtained from the MDAT box in the input) using the same key string, IV and AES-128 CTR mode. The output hex string from the online encryptor matches the same byte area in the output file. Specifically, the online AES encryptor encrypted the highlighted hex string “<em>72 6f 92 da 51 53 0d ac 8c</em>” in Figure 7 to “<em>19 2f 8b 0f 21 21 8a c8 c1</em>” in Figure 8, and this matches the encrypted hex output of my program which is highlighted in Figure 9.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*LaC-VByOB2zIg_BbVjR4-g.png" /><figcaption>Figure 7: Clear content in MDAT box (highlighted part) of the input file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QFeapQOlKSlwyQZdtPssiw.png" /><figcaption>Figure 8: AES-128-CTR encryption result</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*P9JrI0wWJVeP8zIbJ1eQ8Q.png" /><figcaption>Figure 9: Encrypted content in MDAT box (highlighted part) of the output file</figcaption></figure><p>Note that, in the output file, all the MP4 boxes are unencrypted except for MDAT which stores the media samples. Within MDAT, the first few bytes are also unencrypted which are box type, size, start and probably the elementary stream header of the first media sample (e.g., the NALU headers and slice headers in a H.264/265 video sample). The hex string I selected in the above example is actually the first few bytes in the entire input file that need to be encrypted.</p><p>That’s all about my demo. If you have any questions or comments, please feel free to reach out to me at maxutility2011@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5076154c074f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up your low-latency HLS server to stream from any source inputs]]></title>
            <link>https://bozhang-26963.medium.com/setting-up-your-low-latency-hls-server-to-stream-from-any-source-inputs-de1e757a6688?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/de1e757a6688</guid>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Mon, 15 Mar 2021 21:54:53 GMT</pubDate>
            <atom:updated>2021-06-28T15:27:33.405Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up your own low-latency HLS server to stream from any source inputs</h3><p>This tutorial is about setting up a Low-Latency HLS streaming server using <strong>FFmpeg</strong>, Apple’s <strong>mediastreamsegmenter</strong> and <strong>NGINX</strong>. Apple’s HLS reference tools ([1]) provides instruction on how to run a simple LL-HLS server using <strong>tsrecompressor</strong> and mediastreamsegmenter. However, this server setup only allows live source streams generated locally from a fixed test pattern. Only up to 4 renditions with fixed bitrates (0.5 Mbps, 2 Mbps, 4 Mbps and 7.5 Mbps) and resolutions are allowed (see Figure 1 below). The tsrecompressor manual mentioned a command line option “-i / — input-file” which generates source streams from a local file. However, tsrecompressor always crashes when this option is used. Though this simple reference server provides a good starting point for LL-HLS evaluation, it cannot be used for more advanced scenarios, for example when developers want to use their own test sources. This tutorial shows how to set up an LL-HLS server that can stream from virtually any live sources, such as webcam, screen capturing, local files or remote sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kQszWvBjl7d-42pB" /></figure><p>Figure 1: An LL-HLS stream generated using Apple’s reference HLS tools</p><p>No proprietary tools are needed for this tutorial. The tools you will need include,</p><ul><li>Apple’s HLS stream packager, mediastreamsegmenter,</li><li>Apple’s PHP script for low-latency HLS serving, <strong>lowLatencyHLS.php</strong>,</li><li>FFmpeg (with the basic build) is used to generate live source streams,</li><li>NGINX web server for LL-HLS delivery,</li><li><strong>PHP-FPM</strong> (fastCGI process manager) is needed to run lowLatencyHLS.php,</li><li>Testing players: <strong>Shaka player</strong>, <strong>Hls.js</strong>, <strong>Theoplayer</strong> and <strong>AVPlayer</strong>.</li></ul><p>This tutorial starts by reviewing how to set up a simple LL-HLS server using Apple’s reference HLS tools. Next, it describes a more complex setup with arbitrary live sources and video encoding profiles.</p><p>This tutorial requires high-level understanding of the LL-HLS standard ([2]).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2UwpVuzPd-7Qddtd" /></figure><p>Figure 2: Multi-bitrates LL-HLS streaming from FFmpeg-generated live sources, played with low-latency on Shaka player</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/0*nHUkIQ6VuXsRq0pm" /></figure><p>Figure 3: LL-HLS master playlist with 4 renditions</p><h3>Apple’s reference LL-HLS tools</h3><p>Apple provides the required tools ([1]) for setting up a reference LL-HLS server. You can login to your Apple developer account and download the tools as a tar file. After installation, you can follow the readme file contained in the installation folder to set up the server. Basically, developers can use tsrecompressor to generate live source streams, and send the streams to the HLS packager, mediastreamsegmenter. Tsrecompressor can generate up to 4 live source streams that are time-synchronized. To ingest the source streams, four instances of mediastreamsegmenter need to run, each instance handles one source stream. The live source streams are transported from tsrecompressor to mediastreasegmenter(s) as MPEG Transport Stream (TS) over multicast UDP. Mediastreamsegmenter receives the source streams, repackages them to one of the three formats, TS, CMAF, or FMP4, then outputs LL-HLS chunks/segments and playlists. The LL-HLS output can be hosted on a web server, such as NGINX or Apache server. In addition, since an LL-HLS stream is essentially dynamic web content, the reference server also includes a PHP script, lowLatencyHLS.php to serve the stream chunk by chunk as they are dynamically generated.</p><p>Tsrecompressor can only generate up to 4 live source streams from a fixed test pattern (Figure 1). The live source streams are constrained to a set of fixed bitrates and resolutions. The rest of this article describes how to set up a more advanced LL-HLS server that can serve LL-HLS streams from virtually any live sources. This server setup has been tested against 4 LL-HLS players, Shaka player, Hls.js, Theoplayer and AVPlayer.</p><h3>Setting up the origin streaming server</h3><p>NGINX ([3]) can be used for delivering LL-HLS streams. However, you can also use other web servers. You will need to enable the following NGINX features as they are needed by LL-HLS delivery,</p><ul><li>Running PHP script in NGINX for serving streams in low latency mode.</li><li>HTTP/2: Even though in the latest LL-HLS specification, Apple dropped the requirement for HTTP/2 push, it is still recommended according to Appendix B.1. ([2]).</li><li>HTTPS: Because of HTTP/2, HTTPS is also needed subsequently as most web browsers require encryption for HTTP/2 traffics.</li></ul><p>Depending on your specific setup, you might want to apply other NGINX configurations. For example, the location directory of the server must match the output directory of mediastreamsegmenter, you might also need to configure CORS (cross-origin access) depending on where you run the players.</p><h3>Setting up PHP-FPM and connecting it to NGINX</h3><p>To run PHP scripts, you need to install and run PHP-FPM ([4]) first, then configure NGINX to talk to PHP-FPM. PHP-FPM can be run from anywhere reachable from the NGINX server. For simplicity, you can run PHP-FPM from the same machine as NGINX. The following NGINX configuration shall be added,</p><p>location ~ \.php$ {</p><p>try_files $uri = 404;</p><p>fastcgi_pass 127.0.0.1:9000;</p><p>fastcgi_index index.php;</p><p>fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;</p><p>include fastcgi_params;</p><p>}</p><p>Above, the localhost IP and port 9000 are used by NGINX to reach PHP-FPM. You can configure PHP-FPM to listen on port 9000 by modifying PHP-FPM’s <a href="http://www.conf/">www.conf</a>. Look for a line like “listen = 127.0.0.1:9000”. Restart PHP-FPM after modifying the configuration. Now NGINX should talk to PHP-FPM, lowLatencyHLS.php can be run by PHP-FPM.</p><h3>Configuring NGINX location directory</h3><p>The NGINX location directory (e.g., /tmp/llhls for location “/llhls” in your domain) must match the output directories of mediastreamsegmenter. For example, if mediastreamsegmenter outputs to /tmp/llhls/, you can configure “/tmp/llhls/” to be the home directory for the location “/llhls”.</p><h3>Configuring HTTP/2 in NGINX</h3><p>HTTP/1.1 works well for LL-HLS playback using Shaka player, Hls.js and AVPlayer, however Theoplayer requires HTTP/2 which is recommended per the LL-HLS specification. Enabling HTTP/2 in NGINX requires adding only one directive in the configure file, “listen 443 ssl http2;”. However, since TLS (Transport Layer Security) is required by most browsers when HTTP/2 is used, you will also need to configure HTTPS in NGINX.</p><h3>Configuring HTTPS in NGINX</h3><p>You need to generate and configure the TLS certificate and private key to enable HTTPS. For example, on Linux, you can use openssl to create self-signed certificates. To add a self-signed certificate to NGINX, you need to add the following directives to the server section in the NGINX configuration file,</p><p>ssl_certificate [path_to_your_cert_file];</p><p>ssl_certificate_key [path_to_your_key_file];</p><p>For web players (Shaka, Hls.js and Theoplayer), self-signed certificate is enough for HTTPS-based streaming. However, for AVPlayer on Apple devices, App Transport Security (ATS) does not trust self-signed certificates by default. There are two workarounds. First, in your AVPlayer application code, catch the “untrusted certificate” error, and add an exception by trusting your self-signed certificate. However, the recommended way by Apple is to build your own Certificate Authority (CA), and create a CA-signed certificate ([5]) for your site. Then, on your Apple devices, add and trust your CA root certificate ([6]), so in the future, any certificate signed by your own CA will be trusted by your AVPlayer application. HTTPS-based streaming should succeed subsequently.</p><h3>Running mediastreamsegmenter and lowLatencyHLS.php</h3><p>Mediastreamsegmenter ingests live source streams and uploads LL-HLS segments/chunks and playlists to the web server. For settings where mediastreamsegmenter and the web server run on the same machine, mediastreamsegmenter can dump the LL-HLS segments/playlists directly to the configured location directory in NGINX. For example, you can create a directory called “llhls/” under your NGINX root directory. Then, under llhls/, you need to create multiple sub-directories to hold the LL-HLS segments and playlists, one for each rendition, e.g., “llhls/low”, “llhls/mid”, “llhls/high”. Before running mediastreamsegmenter, you need to copy lowLatencyHLS.php to each of the per-rendition folders (This is covered in the instruction of Apple’s HLS tools [1]). You also need to place the master playlist under “llhls/” and the master playlist shall refer to the right rendition folder for holding LL-HLS segments and variant playlists, so that they can be found by NGINX and run by PHP-FPM.</p><p>The instruction of Apple’s HLS tools ([1]) shows how to start mediastreamsegmenter. You need to specify the right folder for outputting the LL-HLS segments and playlists using the “-f” option. You also need to specify the right multicast address and port for receiving the live source streams. It is better to run mediastreamsegmenter and your live source generator (e.g., FFmpeg) from the same local area network, so as to avoid complex multicast routing configuration. Make sure the firewall does not block the UDP ports for ingestion. Avoid using multicast addresses starting with “224.” as some of them are reserved, anything between 225 and 239 should be good. Finally, you need to specify the output segment formats. The default is MPEG-TS, you can also specify “ — iso-fragmented” (FMP4) or “ — cmaf-fragmented” (CMAF). For playback using Shaka player and Hls.js, FMP4 or CMAF should be used. For AVPlayer, both CMAF and TS should work, but somehow I found it is better to use TS for performance concern. You can run “man mediastreamsegmenter” to view more options. When a mediastreamsegmenter instance starts to run, it waits for live source streams to come in.</p><h3>Running FFmpeg to generate the live source streams</h3><p>FFmpeg can be used to generate the live source streams, and send them to mediastreamsegmenter(s). You can use webcam, screen capturing, repeatedly playing a local file, or receiving from another external live source. For using local files as the sources, you can use “-re -i [file_name]” to read from a local file, and use “-stream_loop -1” to play it repeatedly. Next, you can transcode the source input to multiple renditions in a single FFmpeg command, such that all the renditions are time-synchronized. Unlike tsrecompressor which supports up to 4 renditions with a fixed set of bitrates, FFmpeg is theoretically unlimited in the number and encoding profiles of the renditions. You only need to start the same number of mediastreamsegmenter(s) to handle all the renditions. You can specify the resolutions, bitrates, the output format (e.g., mpegts) and destinations (IPs and ports) for your renditions. However, you need to keep the frame rate and key frame interval fixed for the generated live source, e.g., “-force_key_frames “expr:gte(t,n_forced*1)””, and “-r 30”. Fixed key frame interval is needed to avoid variable segment duration. Players may not play stably with variable segment duration. Note that mediastreamsegmenter has to be running when FFmpeg transcoder starts. Shortly after you start FFmpeg, mediastreamsegmenter will start to ingest and output LL-HLS stream. Finally, re-encoding multiple renditions could be very resource-consuming, make sure your machine has enough power to do the work. If your machine does not possess the power to run both mediastreamsegmenter and FFmpeg, you may run them in different machines. However, mediastreamsegmenter may complain about “misaligned segments”. If this happens, you can configure mediastreamsegmenter and FFmpeg to communicate via an unicast IP address, instead of multicast IP address.</p><h3>Playback</h3><p>This LL-HLS server setup has been tested using 4 video players that support LL-HLS. You can build simple player applications using the player core libraries provided by the players. For web players, you can run the players in Chrome. For Shaka player, make sure you set “lowLatencyMode” and “autoLowLatencyMode“ to true. For Hls.js, make sure you set “lowLatencyMode” to true. For Theoplayer, you may refer to [7] for how to configure LL-HLS playback.</p><p>For AVPlayer, LL-HLS support has been available since iOS 13.3 in 2019. You will need to write a simple iOS player application using the AVFoundation SDKs. Nothing special is needed to enable low-latency mode. Xcode is needed to run the player application on an iPhone. You can also write player applications for MacOS or iPadOS.</p><h3>Troubleshooting</h3><ol><li>If your player does not start because it fails to download the variant playlists (lowLatencyHLS.php), make sure your NGINX can talk to PHP-FPM. Variant playlists can fail to load if the PHP script does not run properly.</li><li>Make sure your LL-HLS master playlist describes the stream and renditions correctly and matches your FFmpeg encoding profiles.</li><li>Make sure your server runs on a machine with enough system resources, because FFmpeg transcoding can consume lots of CPU cycles and memory.</li><li>Make sure HTTPS is properly configured in NGINX and in the players and browsers. Make sure CORS (cross domain access) is properly configured in NGINX. Alternatively, you may also host your (web) players on the same server as the LL-HLS stream.</li><li>For AVPlayer, make sure your CA-signed certificate is used on both client and server side, and CA root certificate is trusted by your Apple devices. Before running AVplayer on real devices, first run it on emulated devices. The error logs and error numbers printed on the Xcode console can also reveal lots of debug information.</li><li>You can use Apple’s mediastreamvalidator ([1]) to validate your LL-HLS stream if playback fails, it should provide lots of hints.</li><li>You can always run your LL-HLS streams first in Safari. Even though Safari won’t play the stream at low latency, it can help trouble-shooting the issues.</li><li>If AVPlayer does not play the stream at low latency, make sure you are running an iOS version greater than 13.3.</li></ol><p>If you have any questions about this tutorial, feel free to contact me at maxutility2011@gmail.com.</p><h3>References</h3><p>[1]. About Apple’s HTTP Live Streaming Tools, <a href="https://developer.apple.com/documentation/http_live_streaming/about_apple_s_http_live_streaming_tools">https://developer.apple.com/documentation/http_live_streaming/about_apple_s_http_live_streaming_tools</a></p><p>[2]. HTTP Live Streaming, 2nd edition, <a href="https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-08">https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-08</a></p><p>[3]. NGINX, <a href="https://github.com/nginx/nginx">https://github.com/nginx/nginx</a></p><p>[4]. Php-fpm: <a href="https://php-fpm.org/">https://php-fpm.org</a></p><p>[5]. Creating Certificates for TLS testing, <a href="https://developer.apple.com/library/archive/technotes/tn2326/_index.html#//apple_ref/doc/uid/DTS40014136">https://developer.apple.com/library/archive/technotes/tn2326/_index.html#//apple_ref/doc/uid/DTS40014136</a></p><p>[6]. Self-Signed certificates in iOS apps, <a href="https://medium.com/collaborne-engineering/self-signed-certificates-in-ios-apps-ff489bf8b96e">https://medium.com/collaborne-engineering/self-signed-certificates-in-ios-apps-ff489bf8b96e</a></p><p>[7]. How to configure THEOplayer to play your low-latency HLS streams, <a href="https://docs.theoplayer.com/how-to-guides/07-miscellaneous/03-low-latency/02-configure-ll-hls.md">https://docs.theoplayer.com/how-to-guides/07-miscellaneous/03-low-latency/02-configure-ll-hls.md</a></p><h3>About the author</h3><p>Bo Zhang is currently a staff video research engineer at Brightcove inc. His area of expertise include online video streaming, IP networking and telecommunications. He received Ph.D. degree from George Mason University, M.S degree from University of Cincinnati, and B.S. degree from Huazhong University of Science and Technology, all in computer science. He received the best paper award from ACM MSWiM 2011 — the flagship research conference of ACM SIGSIM.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de1e757a6688" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing low-latency DASH in Video.js]]></title>
            <link>https://bozhang-26963.medium.com/implementing-low-latency-dash-in-video-js-ea8d715613ca?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea8d715613ca</guid>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Sun, 13 Dec 2020 01:16:02 GMT</pubDate>
            <atom:updated>2021-04-29T02:14:52.119Z</atom:updated>
            <content:encoded><![CDATA[<p>This article shows a proof-of-concept implementation of Low-Latency DASH (LL-DASH) streaming in <a href="https://github.com/videojs">Video.js</a>, a popular open-source media player for live and VoD OTT streaming. To my knowledge, LL-DASH is only supported by a limited number of players, such as the GPAC player, Dash.js and TheoPlayer. So I guess supporting it in Video.js could still be interesting to the community.</p><p>This LL-DASH implementation allows Video.js to achieve <strong>as low as 1 second</strong> latency (Figure 1), and an <strong>average latency of ~2 seconds</strong>. As seen in Figure 1, the DASH media segments are downloaded using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">Fetch</a> API, even though the initialization segments are still downloaded using <a href="https://www.w3schools.com/xml/xml_http.asp">XmlHttpRequest</a> API. A demo LL-DASH Video.js player is available for evaluation at, <a href="https://ec2-34-218-35-44.us-west-2.compute.amazonaws.com/videojs_lowLatencyDash/http-streaming/index.html?debug=false&amp;autoplay=false&amp;muted=false&amp;minified=false&amp;liveui=false&amp;partial=false&amp;url=https%3A%2F%2Fakamaibroadcasteruseast.akamaized.net%2Fcmaf%2Flive%2F657078%2Fakasource%2Fout.mpd&amp;type=application%2Fdash%2Bxml&amp;keysystems=&amp;buffer-water=false&amp;low-latency-dash=true">https://ec2-34-218-35-44.us-west-2.compute.amazonaws.com/videojs_lowLatencyDash/http-streaming/index.html?debug=false&amp;autoplay=false&amp;muted=false&amp;minified=false&amp;liveui=false&amp;partial=false&amp;url=https%3A%2F%2Fakamaibroadcasteruseast.akamaized.net%2Fcmaf%2Flive%2F657078%2Fakasource%2Fout.mpd&amp;type=application%2Fdash%2Bxml&amp;keysystems=&amp;buffer-water=false&amp;low-latency-dash=true</a>. I have only tested using Chrome. If the video does not start, please refresh the link and try again. As I have tested, the demo player works most of time, but can still refuse to play due to time-sensitive cases. The source code of this POC implementation is available in GitHub, https://github.com/maxutility2011/videojs_lowLatencyDash. The main file for implementing low latency segment downloading and appending to MSE source buffers is located in http-streaming/src/video-segment-fetch.js. There are also lots of code here and there to get the video chunks into the source buffers properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KAmJFkJ-PUsRDDmn" /></figure><p>Figure 1: Video.js plays LL-DASH stream with 1 second latency</p><p>In the rest of this article, I will briefly introduce how LL-DASH works in general, then show how I implemented LL-DASH in Video.js.</p><h3>1. Overview of LL-DASH streaming</h3><p>Before LL-DASH, a live video encoder produces a sequence of video segments, each segment is a few seconds long of video data. Every video segment is a closed GOP starting with an IDR frame. Within a video segment, the encoding of one video frame could be dependent on other frames following it. As a result, a client has to receive a full segment, before it can start decoding the segment. As such, the live latency is at least one full segment long. Also, to avoid rebuffering and improve playback stability, some players may have to add additional buffer time which is a few more segments long. This could further increase the latency.</p><p>The key method which LL-DASH uses to achieve low latency is chunk-based video encoding and data transfer. A video <strong>chunk</strong> is a sequence of video frames which do not depend on the following chunks. Each chunk is grouped into a single “moof” and “mdat” atom when it is packaged in ISO-BMFF format. The duration of a video chunk could be as low as one video frame, but is usually several hundreds milliseconds. As soon as the video encoder produces the first chunk of a video segment, the player can start requesting the segment using HTTP/1.1’s <a href="https://learn.akamai.com/en-us/webhelp/media-services-live/media-services-live-encoder-compatibility-testing-and-qualification-guide-v4.0/GUID-A7D10A31-F4BC-49DD-92B2-8A5BA409BAFE.html"><strong>chunked transfer encoding</strong></a>. Using chunked transfer encoding, a player receives a whole segment one chunk by another as they are produced by the video encoder, until the last chunk is ready. Because a downloaded chunk does not depend on any following chunks, it can be decoded and rendered right after it is downloaded. As more chunks become available in that segment, the encoder continues to push the following chunks to the player using chunked transfer encoding, until the end of the segment (signaled by an empty chunk as per HTTP/1.1 chunked transfer encoding). Such chunk-based encoding and transfer reduces the live latency from at least one segment long (a few seconds) to one chunk long (several hundreds milliseconds). All the elements on a live streaming workflow have to support chunk-based encoding/decoding and data transfer in order to achieve low latency streaming in an end to end manner. This includes the video encoder, the origin server, the CDN and players.</p><h3>2. Implementing LL-DASH in Video.js</h3><h4>2.1. Supporting “availabilityTimeOffset”</h4><p>LL-DASH requires the players to support two MPD attributes, “<strong>availabilityTimeOffset</strong>” and “<strong>availabilityTimeComplete</strong>”. The use of “availabilityTimeOffset” is to notify the player to request the segment as soon as the first video chunk becomes available. A DASH live player uses the following equation to adjust the segment availability time, <strong>adjustedSegmentAvailabilityTime</strong>. This is the earliest time which the player can request the segment,</p><h4>adjustedSegmentAvailabilityTime = segmentAvailabilityTime + segmentDuration — availabilityTimeOffset</h4><p>Eq. 1: calculating adjustedSegmentAvailabilityTime</p><p><strong>segmentAvailabilityTime</strong> is the MPD availability time of the segment, <strong>segmentDuration</strong> is the segment duration. According to the DASH specification, “availabilityTimeOffset” specifies an offset to define the adjusted segment availability time”.</p><p>For regular DASH live players, availabilityTimeOffset is defaulted to 0, so the player can only request a segment when that segment becomes fully available, i.e., at the time segmentAvailabilityTime + segmentDuration. By setting availabilityTimeOffset, the player requests the segment at an earlier time, before the segment becomes fully available, segmentAvailabilityTime + segmentDuration - availabilityTimeOffset. An example is given as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/0*jpF0f7rq5j-RMuxA" /></figure><p>Figure 2: segment request time of a LL-DASH player</p><h4>2.2. Using “Fetch” API instead of “XHR” API to download chunked media segments</h4><p>In low latency mode, we shall use <a href="https://flaviocopes.com/stream-api/">fetch</a> API to download a segment chunk by chunk (as ReadableStream), instead of using the xhr API. The fetch API supports reading partial data of an object as if it is read from a stream. From the downloaded partial data, we need to check if it is a valid video chunk by searching for “moof” box and “mdat” box.</p><h4>2.3. Appending the media chunks as soon as they are downloaded</h4><p>When a media chunk is downloaded, it should be passed to the browser’s MSE source buffer as soon as possible. Regular players only pass full segments to MSE, an LL-DASH player should pass media data to MSE more frequently to take advantage of LL-DASH’s chunk-based downloading.</p><h4>2.4. Incorporating availabilityTimeOffset in the computation of player-side segment availability window</h4><p>“availabilityTimeOffset” adjusts the availability time of segments, therefore it affects the calculation of the player-side segment availability window. “availabilityTimeOffset” should be involved in the calculation of segment availability time.</p><p>Finally, if you have any question about this article, please feel free to contact me at maxutility2011@gmail.com.</p><h3>About the author</h3><p>Bo Zhang is currently a staff video research engineer at Brightcove inc. His area of expertise include online video streaming, IP networking and telecommunications. He received Ph.D. degree from George Mason University, M.S degree from University of Cincinnati, and B.S. degree from Huazhong University of Science and Technology, all in computer science. He received the best paper award from ACM MSWiM 2011 — the flagship research conference of ACM SIGSIM.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea8d715613ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reducing Streaming Startup Delay Using HTTP/2 Server Push]]></title>
            <link>https://bozhang-26963.medium.com/reducing-streaming-startup-delay-using-http-2-server-push-5fc50e8fae8f?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fc50e8fae8f</guid>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Tue, 27 Oct 2020 21:45:25 GMT</pubDate>
            <atom:updated>2020-11-07T15:29:24.062Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reduce Streaming Startup Delay Using HTTP/2 Server Push</h3><h3>Overview</h3><p>This article describes a server-side only method to reduce streaming startup delay by using HTTP/2 server push. In the context of OTT video streaming, the startup delay is defined as the time interval between when a viewer clicks the “play” button on the video player UI, and when she/he sees the first video picture on the screen. Depending on the network connectivity between a viewer and the CDN, OTT streaming startup delay can be as high as 2 seconds for VoD streaming. For live streaming, the delay could be even higher due to the wait time to join a live stream from a stream access point.</p><p>This article aims at reducing streaming startup delay by minimizing round-trips for downloading the required data before a player can render the first video frame. This is achieved by preemptively pushing all the leading files using HTTP/2 server push, before a video player requests them. A simple h2-push video server is implemented and available on github, <a href="https://github.com/maxutility2011/h2_push_video_server">https://github.com/maxutility2011/h2_push_video_server</a>. No player side change is needed. Through experiments, I observed about <strong>1 second reduction in startup delay</strong>.</p><h3>Optimizing streaming experience using HTTP/2</h3><p>HTTP/2 ([1]) has been around since 2015. It proposes several major improvements over HTTP/1.1 such as,</p><ul><li>Multiplexing multiple requests on a same TCP connection to increase throughput,</li><li>Using HTTP header compression and binary protocol syntax to achieve higher protocol efficiency,</li><li>Using HTTP server push to reduce web content load time.</li></ul><p>HTTP/2 has been an enormous success since inception. In 2020, one third of the Internet traffic is carried in HTTP/2, up from 24% in 2019. All the Google services now run on HTTP/2.</p><p>Even though HTTP/2 can dramatically improve web surfing experience, it does not immediately improve video streaming experience, unless we take advantage of the new features in the server and player implementations. For example, we can modify the players to download multiple media segments in parallel to take advantage of requests multiplexing, in order to improve throughput. However, this requires some non-trivial work of modifying the player implementation to handle parallel downloads, but sequential buffering and rendering of multiple media segments.</p><p>A much simpler optimization is to take advantage of server push to reduce streaming startup delay. The basic idea is for the server to preemptively push the init segment and the first media segment to a player when it requests the manifest file. So the player does not have to request the two files before it can start rendering the first video frame. This could save at least 2 round trips which would otherwise be needed for transferring the files.</p><p>One question people may ask is which rendition (containing the init and the first media segment) should the server push to the client, because the server has no idea how much download bandwidth is available on the player side when the streaming just begins. I found that many player implementations such as ExoPlayer and Video.js, start the playback from the lowest bitrate rendition. As the player obtains more accurate bandwidth measurement and buffers enough media segments, then it can gradually increase bitrate. As a result, in my design, the server pushes segments of the lowest bitrate rendition to the player. Even if the player begins with a higher rendition and discards the pushed segments, the overhead won’t be high given the small segment size in the lowest rendition. It is a good deal to pay a few hundreds kilobytes extra download for 1 second delay saving. Furthermore, if the server ends up requesting segments from a different rendition, the segments can still be requested and downloaded as usual, there is no penalty in delay.</p><h3>Implementation and experiment setup</h3><p>I implemented a <a href="https://github.com/maxutility2011/h2_push_video_server">simple video server</a> for this purpose using node.js. Node.js natively supports <a href="https://nodejs.org/api/http2.html#http2_http2stream_pushstream_headers_options_callback">HTTP/2 server push</a>. The code is very simple as described in the previous section. The server assumes the use of MPEG-DASH streaming, however other streaming protocols such as Apple-HLS can also be supported with minimal modification.</p><p>In my experiments, I ran the server on an AWS EC2 free-tier instance located in the US-West-2 (Oregon) zone. In my demo, the server serves test videos to players using DASH on port 4433. I use Dash.js player for playing the video, but other DASH players should also work. The player runs in Chrome browser, but any browser that supports HTTP/2 should work too. The player is physically located in Boston area.</p><p>To compare with regular, non-optimized server implementations, I also ran a HTTP/2 Nginx server on port 443 to serve the same test video. Note that, I do not enable server push in Nginx, so this is simply a traditional request/response server. I measure the startup delay using the reported download time in the network dev-tool in Chrome. The total time used to download the DASH MPD, the init audio and video segments, and the first audio and video segments provides a good-enough estimation of the startup delay. This is because the player can only render the first video frame after downloading the whole segment. Any further delay in the browser media engine is orthogonal to my optimization and can be ignored. Figure 1 and 2 below show the startup delay for streams served from my server implementation and served from Nginx. As seen from the network dev-tool in Chrome, the leading segments (audio and video “init.mp4&quot;, audio and video “seg-1.m4f”) are all “pushed” (as indicated under the “initiator” column) to the player.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8fnN0wukZ-bIMKTaVYrcYA.png" /></figure><p>Figure 1: startup latency for optimized stream</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZs_OtWQX78dlRytQ1C2IQ.png" /></figure><p>Figure 2: startup delay for non-optimized stream</p><p>The measured delay for non-optimized stream is 1041 milliseconds (Figure 2), and 104 milliseconds (Figure 1) for optimized stream. This represents a delay saving of 937 milliseconds which is easily perceivable by human eyes. I have run the experiment multiple times, similar results were observed.</p><p>I provide a quick <a href="https://ec2-34-218-35-44.us-west-2.compute.amazonaws.com:4433/TOS/manifest.mpd">demo stream</a> on my AWS EC2 instance. To play the stream, you can use <a href="http://reference.dashif.org/dash.js/v3.1.3/samples/dash-if-reference-player/index.html">Dash.js player</a>. You can simply copy the stream url into the player, and click “load”. You can also play the <a href="https://ec2-34-218-35-44.us-west-2.compute.amazonaws.com/TOS/manifest.mpd">non-optimized stream</a> served by Nginx and compare the difference in delay. Finally, both streams only play for 2 seconds because unfortunately my AWS free-tier instance does not have enough space for storing the whole test video. This is why in Figure 1 and 2, you see “404 not found” for seg-2.m4f and onward.</p><p>If you have any question regarding this article, please contact me at maxutility2011@gmail.com.</p><h3>References</h3><p>[1] IETF RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2), <a href="https://tools.ietf.org/html/rfc7540">https://tools.ietf.org/html/rfc7540</a></p><h3>About the author</h3><p>Bo Zhang is currently a staff video research engineer at Brightcove inc. His area of expertise include online video streaming, IP networking and telecommunications. He received Ph.D. degree from George Mason University, M.S degree from University of Cincinnati, and B.S. degree from Huazhong University of Science and Technology, all in computer science. He received the best paper award from ACM MSWiM 2011 — the flagship research conference of ACM SIGSIM. He is involved in organizations such as DASH industry forum, MPEG (ISO/IEC JTC 1/SC 29/WG 11) and CTA-WAVE.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fc50e8fae8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Low-latency dash streaming using open-source tools]]></title>
            <link>https://bozhang-26963.medium.com/low-latency-dash-streaming-using-open-source-tools-f93142ece69d?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f93142ece69d</guid>
            <category><![CDATA[low-latency-streaming]]></category>
            <category><![CDATA[mpeg-dash]]></category>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Fri, 14 Aug 2020 21:49:28 GMT</pubDate>
            <atom:updated>2023-04-05T00:43:49.623Z</atom:updated>
            <content:encoded><![CDATA[<p>In this week, I had spent a few nights to build a low-latency DASH server that can stream live video in a way that conforms to the <a href="https://dashif.org/docs/CR-Low-Latency-Live-r8.pdf">newly published LL-DASH specification</a>. The server was built completely using open source tools. I also wrote some integration code to glue the different pieces together. In this article, I’m going to show how I built and tested the server.</p><p>In this March, the <a href="https://dashif.org/">DASH Industry Forum (DASH-IF)</a> published a new change request to the DASH IOP guidelines on <a href="https://dashif.org/docs/CR-Low-Latency-Live-r8.pdf">low-latency modes for DASH live streaming</a> (LL-DASH). This standard extension is based on the earlier work on <a href="https://www.etsi.org/deliver/etsi_ts/103200_103299/103285/01.03.01_60/ts_103285v010301p.pdf">DVB low latency DASH</a>. It also proposes several important new features such as Resync elements for fast stream joining, service description for describing how players should operate under low latency mode, as well as the implementation details and recommended settings for video encoders, origin servers, CDNs and players. The full specification is available <a href="https://dashif.org/docs/CR-Low-Latency-Live-r8.pdf">here</a>. The proposed changes have been implemented in ffmpeg and dash.js player. Akamai has created a <a href="https://akamaibroadcasteruseast.akamaized.net/cmaf/live/657078/akasource/out.mpd">sample stream</a> for public testing, it is playable with dash.js.</p><h3>Overview</h3><p>First, I’ll provide an overview of what I have done. On the server side, I used ffmpeg to ingest a RTMP source stream from the OBS studio, and to output LL-DASH manifest and media segments. The output DASH stream is saved to my local disk. To handle player requests and deliver the stream with low latency, I also used a node.js GPAC server called <a href="https://github.com/gpac/node-gpac-dash">node-gpac-dash</a>. Node-gpac-dash was developed around 2015/2016 for DVB-DASH low-latency streaming. However it has most features that I need. It has a web server which can handle player requests, it can serve directly “pullable” contents such as, DASH mpd file and init segments to the player. For video segments which require pushing to the player using HTTP Chunked Transfer Encoding (CTE), node-gpac-dash will read segments chunk by chunk and send to the player.</p><p>Since node-gpac-dash is intended for DVB-DASH low latency, I had to make a few changes to integrate it with the new LL-DASH standard and the ffmpeg specific implementation. For example, the original node-gpac-dash assumes each LL-DASH segment ends with an EODS box that signals segment end. This non-standard box is only carried in media segments generated by the GPAC packager — <a href="https://github.com/gpac/gpac/wiki/MP4Box">MP4Box</a>. When node-gpac-dash detects EODS box while reading a video segment, it ends the ongoing chunked transfer by sending out a zero-length HTTP chunk to the player. Since EODS is a non-standard method, ffmpeg does not support it. Instead of using EODS box, I chose to add a new command line option when running node-gpac-dash to indicate the number of video chunks in a video segment. By indicating chunks/segment, node-gpac-dash would easily know when the end of a segment is reached. I implemented this change in <a href="https://github.com/maxutility2011/node-gpac-dash">my modified node-gpac-dash</a> code, and it worked well. However, you will also need 1 line change in FFmpeg to integrate with node-gpac-dash, I will get into this later in this article.</p><p>On the player side, dash.js has also implemented support for the new LL-DASH standard, I simply use it without further change to play the low-latency stream. I tested my implementation on my local MacBook. I was able to achieve a ~4 seconds latency from glass to glass (including video capturing and encoding in OBS, LL-DASH content generation and output in ffmpeg, and stream delivery to the player).</p><p>In the rest of this article, I’m going to provide all the details including how to configure OBS to generate proper live source, the command options to run ffmpeg for ingesting and outputting LL-DASH content, what specific changes I have made to node-gpac-server to perform LL-DASH streaming, and the test results and analysis. A prerequisite before running all the steps is to synchronize your computer with a standard time server, using for example NTPD. Dash.js player synchronzies its time with <a href="http://time.akamai.com/?iso&amp;ms">http://time.akamai.com/?iso&amp;ms</a>.</p><h3>Use OBS to generate RTMP live source</h3><p>I used OBS studio to capture webcam video and to generate the live RTMP source stream. Here is how I configured OBS.</p><ul><li>First, open OBS studio, click on the “settings” button, the setting window will pop up.</li><li>Click on the “Stream” setting, select “Custom” for “Service”.</li><li>Enter “<em>rtmp://[ffmpeg_listening_ip]:[ffmpeg_listening_port]/live/</em>” into “Server”. This is the ffmpeg ingest endpoint. I choose to use RTMP for live ingest, but please feel free to use other protocols, such as RTP, or WebRTC if you prefer encoders other than OBS. “<em>[ffmpeg_listening_ip]:[ffmpeg_listening_port]</em>” is the IP and port where ffmpeg listens for the incoming stream. <em>[ffmpeg_listening_port] </em>should be 1935 for RTMP by default. Also, please make sure your OBS can reach ffmpeg using the IP/port combination.</li><li>Enter “[your_stream_key]” into “Stream key”. My ffmpeg ingests live source at “<em>rtmp://192.168.50.165:1935/live/app</em>”.</li><li>Click “Output”, on the output sub-window, check “Enable Advanced Encoder Settings” box, some additional encoder settings will show up. I entered “<em>keyint=120</em>” into “Custom Encoder Settings”. This will have OBS to generate 1 key frame for every 120 encoded frames. For a frame rate of 30fps, it means GOP (Group Of Pictures) size of 4 seconds. Why do I need to configure this? It is because ffmpeg complained about unstable segment duration caused by variable key frame interval. For low-latency DASH streaming using segment template, players are very sensitive to segment duration variation, so I had to specify a key frame interval to enforce constant segment duration.</li><li>Finally, for accurately measuring the end to end latency, I had to burn the current timestamp into the video. Please follow <a href="https://gist.github.com/sam0737/a0ee8ca253fc5c84b2aa2ac018f7b8ad">this page</a> for how to do this.</li></ul><p>Alternatively, you can also use OBS to stream a local video file in a loop as a live input source. I followed this instruction to set this up, <a href="https://streamshark.io/obs-guide/looping-video">https://streamshark.io/obs-guide/looping-video</a>, it worked well for me. After you are done with all the above configurations, please do not click the “Start Streaming” button right away. You have to run ffmpeg first to listen for the incoming RTMP stream.</p><h3>Use ffmpeg to ingest live source and generate LL-DASH stream</h3><p>First, you will need one line change to FFmpeg. In libavformat/dashenc.c, in function dash_write_packet, find the following line,</p><p>snprintf(os-&gt;temp_path, sizeof(os-&gt;temp_path), use_rename ? “%s.tmp” : “%s”, os-&gt;full_path);</p><p>The above line tells FFmpeg to write partial DASH segments to a temporary file ending with “.tmp” extension, e.g., “chunk-stream0–00001.m4s.tmp”. This temporary file is renamed without the “.tmp” extension when the segment is fully generated, e.g., “chunk-stream0–00001.m4s”. The file renaming can lead to a race condition when the modified node-gpac-dash tries to read the last chunk of a segment at about the same time, and this could crash node-gpac-dash. To avoid the crash, I chose to remove the “.tmp” extension, so that FFmpeg always outputs to the same file no matter it is partially or fully generated, e.g., “chunk-stream0–00001.m4s”. The new line would look like,</p><p>snprintf(os-&gt;temp_path, sizeof(os-&gt;temp_path), use_rename ? “%s” : “%s”, os-&gt;full_path);</p><p>You will need to rebuild FFmpeg after making this change.</p><p>FFmpeg provides a <strong>ldash</strong> option. However, using <strong>ldash</strong> alone will not generate an LL-DASH stream, it has to come together with a bunch of other options. Below is the full FFMPEG command I used,</p><p>“ffmpeg -f flv -listen 1 -i rtmp://192.168.50.165:1935/live/app -c:v h264 -force_key_frames “expr:gte(t,n_forced*4)” -profile:v baseline -an -map v:0 -s:0 320x180 -map v:0 -s:1 384x216 -ldash 1 -streaming 1 -use_template 1 -use_timeline 0 -adaptation_sets “id=0,streams=v id=1,streams=a” -seg_duration 4 -frag_duration 1 -frag_type duration -utc_timing_url “<a href="https://time.akamai.com/?iso">https://time.akamai.com/?iso</a>” -window_size 15 -extra_window_size 15 -remove_at_exit 1 -f dash /usr/local/var/www/ldash/1.mpd”.</p><p>On the ingest side, the input source is a RTMP stream, so I used “<em>-f flv</em>”, “<em>-listen 1</em>” and “<em>-i rtmp://192.168.50.165:1935/live/app</em>”. “<em>-f flv</em>” tells ffmpeg to expect a RTMP (flv) stream. “<em>-listen 1</em>” tells ffmpeg to operate as a RTMP server. “<em>-i rtmp://192.168.50.165:1935/live/app</em>” specifies the RTMP ingest endpoint.</p><p>We can re-transcode the input source into multiple resolutions and bitrates. The transcoding is done by adding a set of “-map” options followed by the video resolution and bitrate.</p><p>Next, the series of options from “<em>-ldash</em>” to the end are for low latency DASH output. “<em>-ldash 1</em>” indicates DASH low latency mode. “<em>-streaming 1</em>” must be specified to use chunked streaming mode for the output. “<em>-use_template = 1</em>” and “<em>-use_timeline 0</em>” must be specified to use segment template to publish segments, otherwise segment timeline will be used which is not supported by LL-DASH. “<em>-seg_duration 4</em>” is set for ffmpeg to use 4 seconds segments. Since I have GOP size configured to 4 seconds in OBS, ffmpeg will package 1 GOP per segment. By default, ffmpeg sets LL-DASH chunk duration to 1 frame/chunk. You may also want to choose your chunk duration. This can be done by using for example “-frag_duration 1 -frag_type duration” option, this will set chunk duration to 1 second. Depending on the segment duration and chunk duration you choose, you can calculate the number of chunks/per segment by dividing segment duration by chunk duration. For example, given 4 seconds segment duration and 1 second chunk duration, each segment then has 4 chunks in it. You will need to specify chunks/segment when you start node-gpac-dash. “<em>-remove_at_exit 1</em>” tells ffmpeg to remove all the DASH output after the process exits, so I don’t need to remove the file manually every time I run ffmpeg. I also added “-window_size 15 -extra_window_size 15” to specify the DASH time-shift buffer depth (the live sliding window) and delete any segments that fall out of the window automatically. This can avoid exhausting your disk space when the server run on your local machine for a long time. To make the stream standard-compliant, you should also specify a timing server as the time source used to timestamp your LL-DASH media segments and manifest. This can be done using the “-utc_timing_url” option, e.g., “-utc_timing_url <a href="https://time.akamai.com/?iso">https://time.akamai.com/?iso</a>”. On the player side, it is recommended that the same time source is used, such that the client is accurately synchronized with the server. For example, Dash.js also uses “<a href="https://time.akamai.com/?iso">https://time.akamai.com/?iso</a>” as the time source. Finally, “<em>-f dash /usr/local/var/www/ldash/1.mpd</em>” specifies the output format and path. For this demo, I chose to write the stream to a local path. FFmpeg will write each segment to the disk in chunked (progressive) manner. You can also upload to a remote path, such as AWS s3, by using “<em>-f dash -method PUT -chunked_post 1 -http_persistent 1 </em><a href="http://bzhangtest.s3-us-west-2.amazonaws.com/"><em>http://your_bucket.amazonaws.com/</em></a>“. The http method for uploading can be “PUT” or “POST”. ”<em>-chunked_post</em>” is used for chunked uploading, AWS S3 supports chunked POST as far as I know, but I have never tried it. “<em>-http_persistent 1</em>“ is specified to use HTTP persistent connection for uploading. You can find more configuration options on the <a href="https://ffmpeg.org/ffmpeg-formats.html#dash-2">ffmpeg format documentation</a>. The <a href="https://dashif.org/docs/CR-Low-Latency-Live-r8.pdf">LL-DASH specification</a> also provides recommended ffmpeg options and parameters for LL-DASH streaming. The following figure shows the content of the DASH mpd file,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K7QKBgTBsARgumGwrt91QA.png" /><figcaption>Figure 1: mpd file generated by ffmpeg</figcaption></figure><p>After you run the full ffmpeg command, please go back to OBS, and click on “Start Streaming”. Ffmpeg will soon start to ingest, repackage and output LL-DASH stream data. Since I choose to write the stream data to my NGINX web root directory, the stream is now ready to play in dash.js player with regular latency. However, it is not ready to stream it with low latency. I still need to run the modified node-gpac-dash to enable low-latency mode with chunked transfer. In the next section, I will show how to do that.</p><h3>Use the modified node-gpac-dash to perform chunked transfer</h3><p>My modified node-gpac-dash code can be found <a href="https://github.com/maxutility2011/node-gpac-dash">here</a>. First, check out the code, copy <a href="https://github.com/maxutility2011/node-gpac-dash/blob/master/gpac-dash.js">gpac-dash.js</a> to one level up of the folder where ffmpeg writes the LL-DASH media segments to. For example, my ffmpeg writes DASH data to “<em>/usr/local/var/www/ldash</em>”, so I copied gpac-dash.js to “<em>/usr/local/var/www/</em>” and ran it from there. Next, run “<em>node gpac-dash.js -chunk-media-segments -cors </em>-chunks-per-segment 4” to start the LL-DASH server. “<em>-chunk-media-segments</em>” must be specified to perform chunked transfer, “<em>-cors</em>” is specified to allow cross-origin. “-chunks-per-segment” specifies the number of chunks in a segment. You will need to use the number of chunks per segment you have used when running FFmpeg. Now I have OBS, ffmpeg and gpac-dash.js all running on my MacBook, these are all I need for running a LL-DASH server. Next, it is time to play the stream with dash.js player.</p><h3>Use dash.js to play the LL-DASH stream</h3><p>Dash.js has all the support of the latest LL-DASH standard, I didn’t make further change to it. I just opened the latest dash.js player (<a href="http://reference.dashif.org/dash.js/v3.1.2/samples/dash-if-reference-player/index.html">v3.1.2</a> as of today) in my browser, and entered the stream URL, “<a href="http://192.168.50.165:8000/ldash/1.mpd"><em>http://192.168.50.165:8000/ldash/1.mpd</em></a>”. Remember to click “Show Options” and check “low latency mode” box. Finally, click “Load” to start playing. When the video shows up, compare the timestamp in the video to your current local time from <a href="http://time.is/">time.is</a>. I found there was a 4 seconds difference, that was my glass to glass delay (Figure 2). At the end of each segment download, node-gpac-dash printed the following line, “end of file reading (ldash/chunk-stream0–00028.m4s) in 3969 ms at UTC 1597419864611”. This means node-gpac-dash serves the segment chunk after chunk in a total of 3969 ms, this matches the duration of the segment transferred. For regular latency DASH, the player downloads segments as fast as the bandwidth allows, the download time would be much shorter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WgNPJ0U2gm1x58eLlDa_Uw.png" /><figcaption>Figure 2: 4 seconds glass to glass delay</figcaption></figure><p>Considering that I run everything from my local machine, 4 seconds delay sounds a bit long for low-latency streaming, but it is still much lower than regular DASH live. Earlier this month, I experimented the community low-latency HLS technology on AWS Interactive Video Service (IVS) using a slightly different setting (OBS capturing from local machine, RTMP ingest into AWS, and playback using a AWS IVS sample player from local machine), the delay was 5 seconds but the server runs in AWS.</p><p>As I measured, it took 2.5~3 seconds from when OBS started capturing and when ffmpeg received the first video frame. This is the ingest delay. By using a faster encoder and real-time ingest protocol such as RTP, I see a possibility of reducing the ingest delay from 2.5~3 seconds to be around 0.5 second, this potentially leads to a glass to glass delay of around 1.5~2 seconds. One more optimization I can do is to save the media segments in memory instead of writing to disk, that may save another several hundred milliseconds.</p><p>OK, that is all about my work on this topic. If you have any question, please reach out to me directly at <a href="http://maxutility2011@gmail.com/">maxutility2011@gmail.com</a>.</p><p><strong>Update — April 2023</strong></p><p>If you experience unstable streaming using Dash.js, it is probably caused by high network jitter between your client and server. You might want to set an “initial delay” in Dash.js as follows,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWzWHzYHVV9RXbwFMINw6g.png" /></figure><p>Or, you can set “-target_latency 8” in the FFmpeg command, e.g.,</p><p><em>ffmpeg -f flv -listen 1 -i rtmp://0.0.0.0:1935/live/app -c:v h264 -force_key_frames “expr:gte(t,n_forced*4)” -r 30 -profile:v baseline -an -map v:0 -s:0 768x432 -b:v:0 949k -maxrate 1557k -bufsize 1557k -map v:0 -s:1 1024x576 -b:v:1 1854k -maxrate 2982K -bufsize 2982K -ldash 1 -target_latency 8 -streaming 1 -use_template 1 -use_timeline 0 -adaptation_sets “id=0,streams=v” -seg_duration 4 -frag_duration 1 -frag_type duration -utc_timing_url “https://time.akamai.com/?iso&quot; -window_size 15 -extra_window_size 15 -remove_at_exit 1 -f dash /home/ubuntu/contents/ldash/1.mpd</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f93142ece69d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A quick latency comparison of Apple LL-HLS and the community-driven LHLS]]></title>
            <link>https://bozhang-26963.medium.com/a-quick-latency-comparison-of-apple-ll-hls-and-the-community-driven-lhls-e3eb3e7447ee?source=rss-c84299d4b33b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3eb3e7447ee</guid>
            <dc:creator><![CDATA[Bo Zhang]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 16:01:43 GMT</pubDate>
            <atom:updated>2020-08-03T04:07:11.636Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quick latency evaluation of Apple LL-HLS and the community-driven LHLS</h4><p>There is an ongoing battle in the field of low latency live video streaming. Basically, 3 technologies (MPEG LL-CMAF, Apple LL-HLS and a community-driven LHLS) are competing for the title of the fastest live streamer. LL-CMAF is a MPEG-led effort on top of the existing low-latency MPEG-DASH technology. Apple’s LL-HLS is another low-latency standard that extends Apple’s HTTP Live Streaming (HLS) standard. There is also a community-driven low latency HLS variant which originates from Twitter’s Periscope platform, and is now led by companies such as Twitch and its parent company AWS, and JWPlayer. AWS recently announced a new Interactive Video Service (IVS) which is built with the Community HLS technology. In this article, I’m going to show the results of a quick latency evaluation of Apple’s LL-HLS and the Community Low-latency HLS (CL-HLS). All the tests were built with off-the-shelf tools. Also, I tried to run the comparison as fair as possible for the two technologies. For example, both test streams are hosted in the same CDN network in the same geo-location. The players were also run from the same wifi network back to back in time. However, since both technologies are new to the industry, my tests were limited by the tools that are available to me. Personally, I don’t have a preference between the two, so my tests are not intended to show one is superior over the other.</p><h3>Technical overview</h3><p>In 2016, Twitter’s Periscope platform introduced the first version of low-latency HLS. Since this initial effort, the live video streaming community has started to collaborate on experimenting different technologies to build low latency streaming, such as chunked video encoding, HTTP 1.1 chunked transfer encoding. This eventually led to a draft of the community-driven “LHLS” specification, published in Aug 2018. However, in June 2019, Apple has announced its own version of Low-Latency HLS (LL-HLS) protocol, which was subsequently turned into an IETF standard.</p><p>The Community LHLS shares similar design principles with LL-CMAF and Apple LL-HLS. It uses chunked encoding (video segments are divided into CMAF chunks or partial TS segments) and HTTP Chunked Transfer Encoding (CTE) to enable early segments delivery before whole segments are generated. HTTP CTE is a feature used by servers to push large, premature data objects of unknown size (such as video segments that are being generated by encoders) to clients in piecemeal manner. The CTE-based data transfer ends when the server sends out a zero-length chunk to indicate no more data will arrive. Note that CTE is only used by LL-CMAF and the CL-HLS. Apple LL-HLS uses the new <a href="https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-07#section-4.4.5.1.3">EXT-X-PRELOAD-HINT</a> tag (a standard HLS tag) to enable early segment delivery.</p><h3>Prefetch video segments in CL-HLS</h3><p>The CL-HLS uses a non-standard tag, EXT-X-PREFETCH to publish in-progress video segments for clients to prefetch. These segments are known as prefetch segments in CL-HLS standard. <strong>Prefetch segments</strong> are partial segments that are published early, even before they are fully generated by encoders. A small number of prefetch segments (usually no more than two) near the live edge are published under EXT-X-PREFETCH tags. It format is:</p><p>#EXT-X-PREFETCH: &lt;URI&gt;</p><p>In the manifest, there are also complete segments preceding the prefetch segments. <strong>Complete segments</strong> are fully-generated and fully-available segments. A prefetch segment becomes a complete segment after N target segment durations, N is the number of prefetch segments allowed by the server. A CL-HLS client should always request the last prefetch segment — the segment that is being generated on the server side, in order to play the stream at low latency. CL-HLS is fully backward-compatible in the sense that a legacy HLS player can ignore the prefetch segments, and request the complete segments that are earlier in the manifest, in order to play at regular latency.</p><h3>Comparison of CL-HLS and Apple LL-HLS technologies</h3><p>In this section I will compare CL-HLS with Apple’s LL-HLS standard extension of HLS. This will include general observations on complexity and level of sophistication of each technology and also several test results.</p><h3>General observations</h3><p>CL-HLS is a simple extension of HLS specification. First of all, chunked video encoding and HTTP chunked transfer encoding are widely available technologies. Second, it adds only minimum changes on top of the standard HLS specifications, such as EXT-X-PREFETCH.</p><p>Apple LL-HLS is a more complicated standard but with more features. For example, Apple LL-HLS requires enumerating all the chunks info in the manifest which exposes the internal structure of segments to clients. This provides potentially more room of optimizations on the client side. For example, since a player has to explicitly request each chunk in a segment, it can accurately measure the download bandwidth and adapt bitrate accordingly. For CL-HLS and LL-CMAF which use HTTP CTE, it would be harder to measure download bandwidth for individual chunks.</p><h3>Test 1 using Twitch.tv</h3><p>I set up a test stream of CL-HLS on Twitch.tv and evaluated it. Specifically, I created a Twitch live channel using OBS camera capture as live input. The OBS captured stream is pushed to Twitch using RTMP. I played the stream using the Twitch player — a web player embedded in Twitch’s web UI. The glass-to-glass delay is around 5 seconds when played from the suburb of Boston.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HReTkSniWk4dWrmEpP4_Zw.png" /></figure><p>As seen in Figure 1, delay was 5 seconds (13:50:29pm EST — 13:50:34pm EST). The video went from Boston to Twitch’s edge servers near Wichita, Kansas (geo-location found by looking-up the Twitch media segment edge server’s <a href="http://video-edge-18e99c.jfk04.no-abs.hls.ttvnw.net/">hostname</a>), then back to Boston. The Twitch player runs in Chrome browser on a Mac computer. The same test was also run on Chrome/Windows setting, latency was also 5 seconds.</p><p>Here is the test instruction. First, open <a href="https://www.twitch.tv/">twitch.tv</a> in Chrome browser, sign in with an account of your own. Find out the RTMP stream key in your Twitch channel’s dash board. Next, open OBS, in stream configuration, select “Twitch” under “Service”, then put your Twitch channel’s RTMP stream key in “Stream Key”. Now you can start OBS camera capturing and streaming to your Twitch channel. In another device, open Twitch in browser or in app, find your channel and play it. Finally, open <a href="https://time.is/">time.is</a> to get your current local time, compare to the timestamp shown on the channel video. This gives you the glass-to-glass latency of the channel.</p><h3>Test 2 using AWS Interactive Video Service (IVS)</h3><p>In this experiment, I created my own low latency stream hosted in the same CDN network. I also used low-latency video players to play the streams from the same physical location. I managed to get these done using CL-HLS and Apple LL-HLS streams hosted both in the AWS Northern Virginia zone. I used a web-based player provided by AWS to play the CL-HLS stream, and an iphone 7 running iOS 14 beta to play the Apple LL-HLS stream. Specifically I developed a simple iOS video player using iOS AVPlayer framework. Starting from iOS 14 beta, we no longer need entitlement to play LL-HLS streams. This was recently announced in WWDC 2020, please refer to <a href="https://developer.apple.com/videos/play/wwdc2020/10228/">here</a> for details.</p><p>The tests show that,</p><ul><li>The Apple LL-HLS test with a Wowza sample stream and iOS AVPlayer achieved a glass-to-glass latency of <strong>3 seconds </strong>(15:40:37pm EST — 15:40:40pm EST).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pnnK8pQpNGtn9VBvZp8I3w.png" /></figure><p>In the above figure, please compare the time shown on the computer screen (the current local time from <a href="https://time.is/">time.is</a>), and the burned-in timestamp shown on the iphone screen (the top-left corner).</p><ul><li>The CL-HLS test with AWS IVS stream and IVS sample player achieved a latency of <strong>5 seconds </strong>(15:21:55pm EST — 15:22:00pm EST).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kLQsHNS5sREuxbznJ-gpEA.png" /></figure><p>Since both streams are hosted in the same CDN and in the same geo-location, and both players are also located in the same wifi network in exactly the same geo-location (I was sitting on the same chair when running both tests back to back in time), my tests of CL-HLS and Apple LL-HLS provides a relatively fair comparison of the two competing technologies. <strong>Apple LL-HLS seems to be ~2 seconds faster than CL-HLS. </strong>However I don’t know what protocol Wowza uses to ingest the live input and where the live input comes from. For example, if the live source is also hosted in AWS and the live input is ingested using real-time protocols such as WebRTC, then the ingest delay would be very small. In this case, my comparison would be unfair because my LL-HLS test used off-cloud OBS broadcaster and RTMP for ingestion.</p><p>There are three things that are different for CL-HLS test and Apple LL-HLS test that might have an impact on the results. First, the CL-HLS stream (AWS-IVS) uses TS segments, while the Apple LL-HLS stream (Wowza) uses CMAF segments. However, the media container formats are unlikely to have a big impact on latency. Second, the video chunk duration of CL-HLS is unknown. While chunk duration does affect latency, the impact should be negligible as long as they are both very small (e.g., in the magnitude of sub-500 milliseconds). Also, chunk duration is a configurable encoder parameter, neither CL-HLS nor LL-HLS confines its value. Third, the CL-HLS stream has 4 bitrates while the Apple LL-HLS stream only has 1 bitrate, also the stream bitrates are different. It would be more fair to use the same live input with the same encodes for comparison. However, as of this writing, I can’t find any open-source software for this purpose. In the rest of this section, I provide the detailed instructions on how I implemented the current test design.</p><h3>Instructions for reproducing the results</h3><p>For CL-HLS, I used OBS broadcaster and AWS Interactive Video Service (IVS) to generate the CL-HLS stream. AWS IVS is a newly released AWS service that streams low latency live channels using CL-HLS technology. AWS IVS is currently only available in the AWS Northern Virginia zone.</p><p>The instruction for generating a CL-HLS stream using AWS IVS is given as follows.</p><ol><li>Sign in to your AWS account. I used my own personal account with the free, basic plan.</li></ol><p>2. Search the “AWS IVS” service. The console should look like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ns5k_wXnP9Z5fY-Rak-hdw.png" /></figure><p>3. Click the “Create channel” button on the top right corner. Follow the wizard to create your low-latency channel.</p><p>4. After creating a new IVS channel, find out your RTMP ingest server url and the stream key on the channel info page. You will need to use them later for configuring OBS broadcaster. Also, please copy the “Playback URL”. This is the channel playback URL you will put into the player later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*exGfIEma2Bbb9KLi10WcEA.png" /></figure><p>5. Open OBS broadcaster on your computer. Go to “Settings -&gt; Stream”, and enter your live channel’s ingest server url into “Server”, and stream key into “Stream Key”. Close the OBS “Settings” window, and click the “start streaming” button from the main window to start OBS camera capturing and streaming to AWS IVS ingest endpoint. To measure latency accurately, you might want to burn a timestamp into your captured video. Please follow <a href="https://gist.github.com/sam0737/a0ee8ca253fc5c84b2aa2ac018f7b8ad">this page</a> for how to do so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9SIcBJVH_NHgApAmbg1RQ.png" /></figure><p>6. Now your live channel should be up and running. Open the <a href="https://codepen.io/amazon-ivs/pen/c3b13a2df34b60ada7756f3a2af8d2f0">AWS IVS sample player</a> in your browser (e.g., Chrome). It looks like the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TdnZsHBAEvOG7QC2uNTkgQ.png" /></figure><p>7. On the top-right corner, there is a “JS” section for embedding javascript. Scroll down to the bottom of the script, replace the URL string in function “player.load(…)” with your IVS channel playback URL obtained in step 4. Then click the “Run” button, the playback should start in 2 seconds.</p><p>8. Open <a href="https://time.is/">time.is</a> to get your current local time, and compare to the burned-in timestamp in the video side by side, the difference is the glass-to-glass latency caused by CL-HLS.</p><p>For Apple LL-HLS, I used a <a href="https://5d6e17f1ca731.streamlock.net/LowLatencyBBB/myStream/playlist.m3u8">sample stream</a> from Wowza which happens to be also hosted in AWS Northern Virginia zone (server IP <a href="https://whatismyipaddress.com/">found to be</a> located in Ashburn, VA, the town where AWS Northern Virginia data center is located, see Figure 7 and 8). Next, I connected the iphone with XCode 12 running on a MACBook. You need a USB-C to lightning cable to connect the two devices. To build iOS app against iOS 14 beta, you will have to use XCode 12. I built and installed the AVPlayer application on the iphone, and simply opened the app to play the Wowza test stream. Again, open <a href="https://time.is/">time.is</a> to get the current local time and compare to the embedded timestamp in the video. It gives the glass-to-glass latency of Apple LL-HLS.</p><h3>About the author</h3><p>Bo Zhang is currently a staff video research engineer at Brightcove inc. His area of expertise include online video streaming, IP networking and telecommunications. He received Ph.D. degree from George Mason University, M.S degree from University of Cincinnati, and B.S. degree from Huazhong University of Science and Technology, all in computer science. He received the best paper award from ACM MSWiM 2011 - the flagship research conference of ACM SIGSIM. He is involved in organizations such as DASH industry forum, MPEG (ISO/IEC JTC 1/SC 29/WG 11) and CTA-WAVE.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3eb3e7447ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>