<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikita Schmidt on Medium]]></title>
        <description><![CDATA[Stories by Nikita Schmidt on Medium]]></description>
        <link>https://medium.com/@sk.nik?source=rss-b5719c0e0d28------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3kE45JxwmUYjsDMpoo8-WQ.jpeg</url>
            <title>Stories by Nikita Schmidt on Medium</title>
            <link>https://medium.com/@sk.nik?source=rss-b5719c0e0d28------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:54:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sk.nik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to develop your own Twitch and not die]]></title>
            <link>https://medium.com/@sk.nik/develop-your-own-streaming-videoclaster-based-on-webrtc-2b59edb43873?source=rss-b5719c0e0d28------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b59edb43873</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[streaming-video]]></category>
            <category><![CDATA[webrtc]]></category>
            <category><![CDATA[simulcast]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Nikita Schmidt]]></dc:creator>
            <pubDate>Mon, 09 Oct 2023 12:51:41 GMT</pubDate>
            <atom:updated>2023-10-09T12:51:41.022Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello everyone. My name is Nikita, I am the CTO of a company where I am responsible for the development of a streaming product. Recently, we completed the development of a new solution for video stream broadcasting based on WebRTC and in this article, I want to talk a little about the technical solution, the challenges we faced, how we overcame them, and how we designed the architecture of our solution.</p><p>Before we dive into the details, it is important to note that if streaming is not a key feature of your product, I would recommend considering ready-made solutions available on the market. However, if video streaming is one of the key features in your application where delivering the best user experience is crucial and you want to avoid ending up with a bill that could bankrupt your product at the end of the month, then it might be worth considering your own solution. Of course, there are many factors to evaluate, which go beyond the scope of this article.</p><p>So, currently there are several technologies available for video streaming, each having its own pros and cons. We have settled on WebRTC technology and have been working with it for several years now. We started our product using open source solutions, specifically our video cluster was built based on Janus. It was a great solution to quickly test our hypothesis, business model, and start scaling the product. However, as our product grew, users demanded higher quality content, which led us to begin developing our own solution.</p><p>If you are developing a streaming solution, be prepared for the fact that it is a rather complex field. While working with a Proof of Concept (PoC) locally and establishing a Peer-to-Peer connection, everything may go smoothly. However, as soon as you start attracting users and your PoC evolves into a solution for end-users, that’s when the problems begin.</p><p>Users start coming to you with various devices, different software versions, and different chips on their devices, all of which somehow affect the performance. The number of users increases, and most importantly, your users come from different parts of the world, each with a different quality of internet connection.</p><p>Well, if you still have the desire to create a streaming solution, let’s start by focusing on the key points:</p><ol><li><strong>Architecture</strong>: The solution should be prepared for automatic scalability in case of increased load and should have mechanisms for resilience in the event of problems.</li><li><strong>Video Quality and Adaptive Streaming</strong>: Ensuring high-quality video delivery is crucial to providing users with the best viewing experience. This may involve selecting the appropriate bitrate, optimizing for different devices, and internet connection speeds.</li><li><strong>Network Infrastructure</strong>: The infrastructure must be ready to handle video streaming, with special attention given to network quality between nodes, latency, and packet loss.</li><li><strong>Monitoring</strong>: any production solution is inconceivable without monitoring both the basic technical health metrics of the cluster and the quality metrics of the transmitted video data specific to each user.</li></ol><p>So, taking into account these key points, this will enable the creation of an efficient and reliable solution for video streaming.</p><h3><strong>Architecture</strong></h3><p>Let’s begin by acknowledging that WebRTC was initially designed to create a connection between two endpoint devices (peers) using a signaling server, enabling them to start audio or video streaming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2KNSd3E0JY-ociW68BoEQ.png" /></figure><p>But in order to solve the task of streaming a video stream from a streamer to an unlimited number of viewers, we need to send this stream to millions of users, and the option of Peer-to-Peer is no longer suitable. For this purpose, we will need our own media server, which will receive the stream from the streamer and at the same time distribute it to our viewers. This is how we gradually approach concepts such as Mesh, MCU, and SFU WebRTC architectures and their tradeoffs.</p><p><strong>MESH</strong>: This is the simplest method that utilizes a peer-to-peer network. Essentially, each participant directly sends their stream to every other participant, bypassing the need for a media server. Each participant also receives streams from all other users. So, if there are 3 participants, you will send your stream to the other two users while also receiving all of their streams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ysFsB47TI8xvJoiJMzQj7g.png" /></figure><p><strong>MCU</strong> stands for Multipoint Control Unit. MCUs support a greater number of participants compared to MESH. Each participant sends their stream to the server, where the server decodes each participant’s stream and creates a new stream from each of the received streams. Finally, the server sends this stream back to all participants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2UyrssTosWB9DFRKMe6WTg.png" /></figure><p><strong>SFU</strong> (Selective Forwarding Unit). This is a more popular approach than MCU when scaling to more than 4–6 participants. This approach also involves a server. The difference is that each participant sends their stream to the server, and the server forwards each received stream back to all the other participants. The great thing about SFU is its scalability, which gives you more flexibility and possibilities.</p><p>To accomplish our goal, we have chosen to use SFU as the solution. But what if our stream becomes very popular? It can happen very quickly, and our single server may not be able to handle the load, for example, in terms of CPU or network capacity. To address this, we introduce multiple levels for distributing our traffic. We call these intermediate servers relays.</p><p>Great, now our streamer connects to the upper level and uses WebRTC to transmit their stream onto it. Then, this server forwards it to the input relays, which can in turn distribute traffic to the next levels. The number of levels can be vary, but it is important to understand that each level adds overhead, some delays, and so on. Therefore, the number of levels should be chosen based on the scaling scheme and the number of users. The last relay level eventually sends the stream to an Edge server, which serves the viewers. By implementing such an architecture, we address the scalability issue of our solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EjkgqtVnJzZDIm9yZCDtBw.png" /></figure><p>Now we need to address another issue. If your servers are deployed in New York and you have a popular streamer who has moved to Thailand, and to make matters worse, your marketing department decided to target advertising to a new region, such as Turkey, your users will once again be unhappy. The problem is that when streaming traffic is routed to a server located far away from the end users, delays start to increase and quality decreases. To solve this issue, we need to deploy our servers in the regions where our users are located and connect these servers with a reliable internet channel. This way, users will transmit and receive media streams from servers closest to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xtl_PPTJHtKAH3wQQmPE-w.png" /></figure><p>Great, we have obtained the basic architecture of our video cluster, which allows us to receive and deliver media streams in close proximity to our users, as well as scale with the growing popularity of our streams. However, we still cannot claim that we have achieved the perfect solution. Why? So far, we have only focused on the scalability issue, while another equally important problem is the resilience of our solution. As long as everything is going well and our servers are handling the traffic for our users, we have no issues. However, at any moment, any of our video cluster machines may fail, and part or all of the viewers will lose sight of their favorite stream.</p><p>To address this challenge, our cluster has service discovery and continuous monitoring. We use serf (which is based on the GOSSIP protocol) for detecting changes in the state of our cluster nodes. At the same time, our video cluster has master nodes that handle state changes and can reconfigure the routes of video streams. They also monitor the load in each region at every level and make decisions regarding horizontal scaling or traffic redistribution between nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5a5zlCPN9HwwCTCpjeC4Jg.png" /></figure><p>Well, we have covered some topics such as architecture, scalability, and fault tolerance. Next, I would like to delve into the video component of our solution.</p><h3><strong>Video Quality and Adaptive Streaming</strong></h3><p>WebRTC is essentially a method and set of technologies for streaming media. In order to delve into them, one would have to study the specifications of each technology used. As for video codecs, WebRTC primarily supports a specific set of codecs, one of which is VP8 and Opus. These are the codecs we use in our solution. There is both hardware and software codec support, but we will not delve into that. Frames are captured and encoded from the device, and then sent using the RTP protocol, typically over UDP. This is a good time to discuss the quality of the video stream and the potential for improvement for different users.</p><p>If you have users, then you definitely have a problem with the fact that each of these users has different internet connection bandwidth. And if our streamer can send videos in Full HD resolution, it’s certain that not all viewers will be able to receive such a data stream. On the receiver side, there will be congestion, packets will be lost, delayed, and the image will have a large number of artifacts. To solve this problem, we will start with Simulcast.</p><p>Simulcast is a feature that allows the streamer to configure the encoder to encode and send the video stream to our server in multiple resolutions. Essentially, this is very similar to the Spatial layer in SVC, which we will discuss later. However, since VP8 has limited SVC support (only temporal layer), we use simulcast. So, on the streamer’s side, we set the encoder parameters to start encoding and transmitting the video in different resolutions, such as 180p, 360p, and 720p. It is also worth noting that problems may arise on the streamer’s side when they are unable to transmit all the layers and need to disable some of them. In WebRTC itself, there are no mechanisms for automatic layer enabling or disabling. Therefore, this is done on the streamer’s side based on feedback received through the RTCP protocol.</p><p>The general scheme looks as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QadZ-cQFKATONcgJ9DQDUw.png" /></figure><p>Here, the streamer sends multiple layers of media data to our server, which receives them and further transmits them to relays. The edge servers then determine which layer needs to be sent to each specific user.</p><p>When it comes to feedback, there are various methods for assessing network throughput, such as REMB, TWCC (GCC). They employ different approaches and algorithms for estimation.</p><p><strong>REMB</strong> stands for Receiver Estimated Maximum Bitrate. This is an RTCP message that uses bandwidth control to prevent network congestion. The REMB is calculated by gathering information about the received packets and their sizes on the receiver side. This information is used to determine the maximum bit rate that the receiver can accept without compromising quality or performance.</p><p><strong>TWCC</strong> — Transport Wide Congestion Control is the latest development in RTCP network status communication. TWCC is almost a hybrid approach between the SR/RR and REMB generations of protocols. TWCC Webrtc is calculated based on several factors. The main ones include: latency, packet loss, jitter. Based on these parameters, TWCC Webrtc calculates statistics and metrics such as end-to-end delay, packet loss, and connection quality.</p><p><strong>GCC</strong> (Google Congestion Control) — The algorithm allowing to estimate the available bandwidth, GCC focuses on packet losses and variations in frame arrival time as two main metrics. GCC tries to figure out if network links are growing larger and larger queue depths before packet loss actually occurs.</p><p>To learn more about the work of all these approaches, I would recommend studying an excellent resource on this topic: <a href="https://webrtcforthecurious.com/docs/06-media-communication/">https://webrtcforthecurious.com/docs/06-media-communication/</a></p><p>When implementing, we tried different approaches, and the TWCC option seems more promising, especially when combined with GCC, as REMB does not always perform well and sometimes it takes considerable time to make a decision on layer switching.</p><p>Well, we have implemented Simulcast and implemented mechanisms for evaluating internet connections and managing the enabling/disabling of layers on the streamer side, as well as their switching on the server side for viewers. But can we further improve? Definitely yes. The next step is to introduce SVC.</p><p>SVC (Scalable Video Coding) is a method that allows encoding a video stream into multiple layers. SVC includes two types of layers: Spatial and Temporal. The encoding of these layers is done in a way that eliminates the need for server-side re-encoding, but instead, specific packets can simply be dropped to reduce resolution or fps on the viewer’s side. The Spatial layer handles resolution, while the temporal layer handles FPS. For example, using L1T3 means having only 1 spatial layer and 3 temporal layers. Therefore, if a viewer is struggling with the stream, we can lower the FPS for them. For instance, if the streamer is originally sending 60 FPS, we can offer 60, 30, or 15 FPS. Sounds great, doesn’t it?</p><p>As I mentioned before, the VP8 codec only supports temporal layers. There are codecs that have full support for both types of layers, such as VP9. However, not all platforms support VP9 yet. Hopefully, this will change in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jkwwtjK1aGU5dGbBr4sMJg.png" /></figure><p>It is important to note that even though we do not need to recode the video stream, we cannot simply discard packets with certain layers. Each VP8 packet contains metadata in its headers, such as TL0PICIDX (8 bits temporal level zero index), PictureID (image identifier, 7 or 15 bits), and KEYIDX (5 bits temporal key frame index). Additionally, we have RTP packet headers that include a Sequence number and Timestamp for each packet. According to specifications, all this metadata must be formatted in a certain way, and on the server side, we need to take this into account. If we discard packets and the specification requires certain headers to always be monotonically changing, we need to rewrite these headers ourselves. The WebRTC specification does not provide clear rules on how to do this, and there are several approaches. However, it is necessary to do so if we want to prevent the user from experiencing video frames loss or an increase in packet retransmission requests due to gaps in the SN RTP sequence when switching.</p><p>In addition to adaptive quality, it is important for us to provide the ability for instant streaming switching between different streamers on the viewer’s end. If you’ve ever used the TikTok app, you’ve surely noticed how quickly swiping and launching the next video happens there. It’s worth noting that TikTok uses a different technology from WebRTC, which makes this process somewhat easier. However, solving this task is still possible. Video codecs have complex algorithms and numerous optimizations. When streaming a video, the video stream consists of key frames and delta frames. This is done to save bandwidth and avoid constantly transmitting complete frames. Usually, a key frame is sent first, followed by frames with only changes. And this is where the main challenge lies — the receiver won’t be able to decode the new stream from the streamer if it doesn’t start from a key frame. Therefore, to solve the issue of instant switching, we always start sending the new video stream from a key frame. This allows the receiver to quickly decode the stream and start displaying the video from the next streamer.</p><h3><strong>Monitoring</strong></h3><p>So, we have discussed many key aspects that need to be addressed when implementing a high-quality media cluster. However, another important part of this solution is monitoring. As I mentioned before, when you have users from all over the world, they all have different conditions for receiving traffic, different network conditions, etc. This is where high-quality monitoring becomes necessary. It should not only track parameters of your video cluster such as CPU, Memory, Network, latency, and others, but also allow you to understand what happened at a particular moment on the streamer’s or viewer’s side, what happened to their connection, how many packets were lost, what the jitter buffer was like, and much more. Without this knowledge, your first line support will be overwhelmed with complaints from frustrated users, and you won’t be able to do anything about it. It will also prevent you from improving your solution and identifying weak links in the chain.</p><p>P.S. In this article, I have attempted to cover the fundamental aspects that one would have to go through when implementing a streaming video solution. This is a brief overview of the key points, and for a more detailed exploration of the details, separate posts would be required for each of them. Currently, we are in the testing and stabilization phase, and in the near future, we plan to begin migrating our products to the new solution. If you have any questions or comments, I am open to discussion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b59edb43873" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>