<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sudeep Dasgupta on Medium]]></title>
        <description><![CDATA[Stories by Sudeep Dasgupta on Medium]]></description>
        <link>https://medium.com/@sudeepdasgupta?source=rss-85a15b01d7ce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Lw-MJ7PDQeTBy9qW1eAXcA.jpeg</url>
            <title>Stories by Sudeep Dasgupta on Medium</title>
            <link>https://medium.com/@sudeepdasgupta?source=rss-85a15b01d7ce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:24:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sudeepdasgupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Broadcasting and multicasting millions of clients using TCP]]></title>
            <link>https://sudeepdasgupta.medium.com/broadcasting-and-multicasting-millions-of-clients-using-tcp-5794d784829a?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/5794d784829a</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[tcp]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[bytearray]]></category>
            <category><![CDATA[bytebuffer]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Sun, 22 Mar 2020 10:05:37 GMT</pubDate>
            <atom:updated>2020-03-23T09:35:53.486Z</atom:updated>
            <content:encoded><![CDATA[<p>TCP is unicast then how can we broadcast in an optimal way. This blog is all about broadcasting in TCP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*LZcnyt_cGVUOKBHXt840uw.png" /></figure><p>In socket programming generally, two protocols are widely used that is TCP and UDP. Above these protocols, more protocols are invented like WebSocket, HTTP, STP, RTMP etc. TCP is generally used where data loss cannot be accepted as TCP made threeway handshake where it sends acknowledgement on every packet it receives but this is way to slow than UDP. UDP is generally used in video streaming where data loss is not a big problem but speed is highly required. UDP does not send an acknowledgement but is much faster than TCP.</p><p>TCP is generally unicast that means the server needs to send the response to each client individually whereas UDP has multicast and broadcast support. Multicast means server transmits data to a group and client connected to the same group gets the data. Broadcast means n number of clients receiving same data from the server. Below is a sample image to show you a relation of on to one, one to many, one to all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*0ElK35yD7FV7bFNsPpv8GA.jpeg" /></figure><p>Now why TCP broadcasting is necessary and even multicasting. Well, suppose you created a WebSocket server or any simple plain TCP server and millions of clients are connected to a single channel for like live streaming of a channel going on and people connected to that channels are writing comments now web-browser does not support UDP yet, WebRTC is based on UDP but is a caller callee protocol. So we will talk about browser connected to a WebSocket server and pushing messages concurrently and the messages need to be shown back to the clients in the UI of other clients who are writing comments or whatever in that case what server does is it creates a hashmap of the array and put all clients socket object inside the array and can map the array to a hashmap using a key as channel name and whenever a message is received it iterates over the array and send the message to all socket that is in the array. Given below a simple example. It will be in Golang but you can correlate with other languages.</p><pre>// Host TCP server in localhost with port 8100<br><br>server, err := net.Listen(&quot;tcp&quot;, &quot;127.0.0.1:8100&quot;)<br> <br>// create array of tcp socket client<br>var listOfClients []*net.TCPConn<br><br>// waiting for new connections<br>for {<br>  conn, err := server.Accept()<br>  <br>  tcp := conn.(*net.TCPConn)<br><br>  tcp.SetNoDelay(true)<br>  tcp.SetKeepAlive(true)<br>  tcp.SetKeepAlive(true)<br>  tcp.SetLinger(1)<br>  tcp.SetReadBuffer(10000)<br>  tcp.SetWriteBuffer(10000)<br>  tcp.SetDeadline(time.Now().Add(1000000 * time.Second))<br>  tcp.SetReadDeadline(time.Now().Add(1000000 * time.Second))<br>  tcp.SetWriteDeadline(time.Now().Add(1000000 * time.Second))<br>  <br>  // appending the client socket to array/slice<br>  listOfClients = append(listOfClients, tcp)<br>  <br>  // calling handleRequest using goroutines to run in seperate thread<br>  go HandleRequest(*tcp)<br>}<br><br>func HandleRequest(conn net.TCPConn){<br>  <br>  // waiting for new messages<br>  <br>  for {<br>      <br>      // reading the message from socket buffer<br>      <br>      message, err := bufio.NewReader(conn).ReadString(&#39;\n&#39;)<br>      <br>      if err != nil {<br>          log.Printf(&quot;Error: %+v&quot;, err.Error())<br>          return<br>      }<br>      <br>      // broadcasting to all clients<br>      go broadCast(message)<br>  }<br>}<br><br><br>func broadCast(message string){<br><br>  // iterating over the listOfClients<br>  <br>  for i := range listOfClients{<br>    // sending message to all clients in the socket<br>    listOfClients[i].Write(message)<br>  }<br>}<br><br>###################################################################################<br><br>// Client Socket example<br><br>// Client A connect to this socket<br><br>// connecting to localhost with port 8100<br>conn, _ := net.Dial(&quot;tcp&quot;, &quot;127.0.0.1:8100&quot;)<br><br>// writing message to socket<br><br>conn.Write([]byte(&quot;Message from client A&quot;))<br><br>for { <br>  // reading messages from server<br>  message, err := bufio.NewReader(conn).ReadString(&#39;\n&#39;)<br>  if err != nil {<br>      log.Printf(&quot;Error: %+v&quot;, err.Error())<br>      return<br>  }<br>  <br>  fmt.Println(message)<br>      <br>}<br><br>  // Same code for another client<br> // Client B connect to this socket<br><br>conn, _ := net.Dial(&quot;tcp&quot;, &quot;127.0.0.1:8100&quot;)<br><br>conn.Write([]byte(&quot;Message from client B&quot;))<br><br>for { <br>  <br>  message, err := bufio.NewReader(conn).ReadString(&#39;\n&#39;)<br>  if err != nil {<br>      log.Printf(&quot;Error: %+v&quot;, err.Error())<br>      return<br>  }<br>  <br>  fmt.Println(message)<br>      <br>}</pre><p>In this example 2 client “Client A and Client B” both of the connected to localhost with port 8100 and send messages to the server. The server receives the message and the send back the message to all clients connected to the server iterating over the socket client array. This is a legacy code used by many developers and you will find such an example in any language over the internet. Now, what is the issue with this example, when the number of client increases the last client socket that is in the array will get the message very lately as it will iterate over the array again and again whenever a new message is received. You may use concurrency for sending all messages in separate threads or creating a batch and then iterate over the socket array and send the batch message after some interval. Still, this is not a better solution.</p><p>So then I came out with a solution this might not be the best solution but I benchmarked it and neither memory utilization is high with n number of client neither CPU and all clients receives message concurrently at the same time without iterating over the socket array over and over again. If you have a better solution please share with me so that I can also learn new stuff from you guys. Below is the code snippet for my solution of the broadcast. I have used a file, but an array can also be used to keep everything in memory, it is up to you but make sure to remove messages from array to avoid memory leaks.</p><pre>// setting the log file where all the messages will be written<br><br>var socketFilePath = &quot;/path/of/the/socket.log&quot;<br>    <br>// opening a file in append mode<br><br>FileDescriptor, err := os.OpenFile(socketFilePath,<br>		os.O_APPEND|os.O_WRONLY, 0700)<br><br>// checking for errors while opening file<br>if err != nil {<br>    go log.Write(err.Error())<br>    return<br>}<br><br>// Host TCP server in localhost with port 8100<br><br>server, err := net.Listen(&quot;tcp&quot;, &quot;127.0.0.1:8100&quot;)<br><br>     <br>// waiting for new connections<br>for {<br>  conn, err := server.Accept()<br>  <br>  tcp := conn.(*net.TCPConn)<br><br>  tcp.SetNoDelay(true)<br>  tcp.SetKeepAlive(true)<br>  tcp.SetKeepAlive(true)<br>  tcp.SetLinger(1)<br>  tcp.SetReadBuffer(10000)<br>  tcp.SetWriteBuffer(10000)<br>  tcp.SetDeadline(time.Now().Add(1000000 * time.Second))<br>  tcp.SetReadDeadline(time.Now().Add(1000000 * time.Second))<br>  tcp.SetWriteDeadline(time.Now().Add(1000000 * time.Second))<br>  <br>  // appending the client socket to array/slice<br>  listOfClients = append(listOfClients, tcp)<br>  <br>  // subscribe to the channel file<br>  go SubscribeChannel(*tcp)<br>  <br>  // calling handleRequest using goroutines to run in seperate thread<br>  go HandleRequest(*tcp)<br>}<br><br>func SubscribeChannel(conn net.TCPConn){<br>    <br>    // opening file <br>    file, err := os.Open(socketFilePath)<br>    <br>    // defering the file to close, it will be closed only when loop is exited<br>    defer file.Close()<br><br>    if err != nil {<br>        go log.Write(err.Error())<br>        return<br>    }<br>          <br>    // creating a file reader<br>    reader := bufio.NewReader(file)<br>    <br>    var line string<br>    <br>    var err error<br>    <br>    var lastModifiedDate int64<br>    <br>    for{<br>        <br>        // getting file status<br>        fileStat, err := os.Stat(socketFilePath)<br><br>        if err != nil {<br>            go log.Write(err.Error())<br>            break<br>        }<br>        <br>        // checking for last modified time<br>        if fileStat.ModTime() == lastModifiedDate{<br>            thread.Sleep(1 * time.Nanosecond)<br>            continue<br>        }<br>        <br>        // reading lines<br>        line, err = reader.ReadString(&#39;\n&#39;)<br>        <br>        if err != nil {<br>            if err == io.EOF {<br>                continue<br>            }else{<br>                break<br>            }<br>        }<br>        <br>        //updating the last modified time<br>        lastModifiedDate = fileStat.ModTime()<br>        <br>        // setting a variable to count number of retry<br>        var totalRetry = 0<br>    <br>        RETRY:<br>        <br>        // if retry is already 3 then exit loop<br>        if totalRetry &gt; 3{<br>            log.Println(&quot;Socket disconnected...&quot;)<br>            break<br>        }<br>        // writing to clients<br>        _, socketError := conn.Write([]byte(line))<br>        <br>        // if failed then retry again till count = 3<br>        if socketError != nil{<br>            totalRetry += 1<br>            goto RETRY<br>        }<br>        <br>    }<br>    <br>}<br><br>func HandleRequest(conn net.TCPConn){<br>  <br>  // waiting for new messages<br>  <br>  var callbackchannels = make(chan bool)<br>  <br>  for {<br>      <br>      // reading the message from socket buffer<br>      <br>      message, err := bufio.NewReader(conn).ReadString(&#39;\n&#39;)<br>      <br>      if err != nil {<br>          log.Printf(&quot;Error: %+v&quot;, err.Error())<br>          return<br>      }<br>      <br>      // broadcasting to all clients<br>      go WriteData(message, callbackchannels)<br>      <br>      &lt;-callbackchannels<br>  }<br>}<br><br><br>func WriteData(message string, callback chan bool){<br>    <br>    mtx.Lock()<br>    <br>    _, err := FileDescriptor.Write(message)<br>    <br>    mtx.Unlock()<br>    <br>    if (err != nil &amp;&amp; err != io.EOF ){<br>        go log.Write(err.Error())<br>        callback &lt;- false<br>        return<br>    }<br>    <br>    callback &lt;- true<br>    <br>}<br><br>###################################################################################<br><br>// Client Socket example<br><br>// Client A connect to this socket<br><br>// connecting to localhost with port 8100<br>conn, _ := net.Dial(&quot;tcp&quot;, &quot;127.0.0.1:8100&quot;)<br><br>// writing message to socket<br><br>conn.Write([]byte(&quot;Message from client A&quot;))<br><br>for { <br>  // reading messages from server<br>  message, err := bufio.NewReader(conn).ReadString(&#39;\n&#39;)<br>  if err != nil {<br>      log.Printf(&quot;Error: %+v&quot;, err.Error())<br>      return<br>  }<br>  <br>  fmt.Println(message)<br>      <br>}<br><br>  // Same code for another client<br> // Client B connect to this socket<br><br>conn, _ := net.Dial(&quot;tcp&quot;, &quot;127.0.0.1:8100&quot;)<br><br>conn.Write([]byte(&quot;Message from client B&quot;))<br><br>for { <br>  <br>  message, err := bufio.NewReader(conn).ReadString(&#39;\n&#39;)<br>  if err != nil {<br>      log.Printf(&quot;Error: %+v&quot;, err.Error())<br>      return<br>  }<br>  <br>  fmt.Println(message)<br>      <br>}</pre><p>This is not the exact solution I use as here you cant track of how much messages the client has already received as in this solution if the client reconnects again it will start downloading all messages from the file secondly you need to set retention mechanism as the file will become larger day by day. Thirdly I prefer byte buffer to send and receive messages where I set first 4 or 8 bytes as the total size of the packet and then add some more header flags in the next byte array then at the end the actual message as it is fast and easy to parse if you JSON it takes a lot of time to serialize and deserialize messages. To read the file I rather use<br><strong>totalByteLen, errPacket := file.ReadAt(restPacket, cursor)</strong> where I update the cursor and keep track of the total bytes sends to the clients and whenever the client has connected again I have the track of the last byte the subscriber received.<br>This byte buffer technique is also used in many stock exchanges for faster delivery, it is also used in WebSocket implementation. I have written a blog where I am given a solution to implement WebSocket from scratch rather than using a library. And this file descriptor solution is also used by many log streaming applications where all messages received by a server are written into a file and then the subscriber receives all the messages that are written in the file. This is way faster than the previous solution where you are iterating over the socket array to send message to all client. Modifying this can also be used as multicast where you create a group using hashmap and put all open subscriber file descriptor that to hashmap and send message to that hashmap as it contains the unique group. I will suggest using byte buffer as using byte buffer message transmission is fast secondly you can send video streams, audio streams etc for realtime communication. You can read the byte buffer library that I have written in Golang create byte buffer in easy steps, check out the other blog written by me. If you have more suggestions and better implementations comment below and suggest solutions. Till then happy coding :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5794d784829a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang Concurrency And Memory Management]]></title>
            <link>https://sudeepdasgupta.medium.com/golang-concurrency-and-memory-management-aab81accdf83?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/aab81accdf83</guid>
            <category><![CDATA[mutex]]></category>
            <category><![CDATA[channel]]></category>
            <category><![CDATA[threads]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[goroutines]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Sat, 14 Mar 2020 11:34:24 GMT</pubDate>
            <atom:updated>2020-03-14T11:42:01.945Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog is about Golang concurrency patterns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*x4Dj3rJHyXgF3ES2VoIQVA.jpeg" /></figure><p>In this blog, we will learn about Golang Concurrency, Channels, Buffered Channels, Mutex, Sync, Context and issues which can occur if we don&#39;t synchronize threads.</p><p>Well, concurrency is running multiple tasks at the same time with shared resources. Concurrency is not parallelism, Parallelism is a completely separate term that we will not discuss here in this blog.</p><p>Concurrency in Golang is created using goroutines. Goroutines are very easy to write using <strong>“go”</strong> keyword just before function.<br><strong>go func(){<br>}()</strong></p><p>Now concurrency helps to complete the task faster than a single thread but too much of concurrency leads to too much of memory consumption as well as CPU utilization.</p><p>Sometimes concurrency leads to deadlock or race conditions. When multiple threads write in a single memory then sequence might get changed. <br>For example, you triggered a thread with a value of 10 and then 20 and you expect the value to be 20 as 20 was triggered after 10 now sometimes due to too much of concurrency you might get value as 10 as 20 might get replaced with 10 this situation is called a race condition. <br>How to get rid of that is simply just you need to synchronize threads. <br>You can use mutex, channels or sync.WaitGroup to synchronize threads.</p><p>We will demonstrate one by one of each one of them.</p><ol><li>sync.WaitGroup will print 5 as all threads are synchronized</li></ol><pre>package main<br><br>package main<br><br>import(<br>	&quot;fmt&quot;<br>	&quot;sync&quot;<br>)<br><br>var number int<br><br>func main(){<br><br><br>	// Here we create a sync.WaitGroup object<br>	var wg sync.WaitGroup<br><br>	// Creating threads adding 1 thread to wait group and sending wait group as parameter of the method<br>	for i := 1; i &lt;= 5; i++ {<br>	    // adding number of threads to wait group<br>	    wg.Add(1)<br>	    // calling worker method with goroutines<br>	    go worker(i, &amp;wg)<br><br>	    wg.Wait()<br>	}<br><br>	// waiting for thread to complete<br><br><br>	fmt.Println(number) // 5<br>}<br><br>func worker(id int, wg *sync.WaitGroup) {<br>	// closing thread after the completion of the method<br>    number = id<br><br>    wg.Done()<br>}</pre><p>2. Mutex, it locks the variable and other threads wait for it to unlock</p><pre>package main  <br>import (  <br>    &quot;fmt&quot;<br>    &quot;sync&quot;<br>)<br><br>var x  = 0 <br><br>func worker(wg *sync.WaitGroup, m *sync.Mutex) {  <br>    m.Lock()<br>    x += 1<br>    m.Unlock()<br>    wg.Done()   <br>}<br><br>func main() {  <br>    var w sync.WaitGroup<br>    var m sync.Mutex<br>    for i := 1; i &lt;= 5; i++ {<br>	w.Add(1)        <br>	go worker(&amp;w, &amp;m)<br>    }<br>    w.Wait()<br>    fmt.Println(&quot;final value of x&quot;, x)<br>}</pre><p>3. Channels, the best feature in Golang to pass messages and synchronize threads. Do not use the same channel in the same thread to read and write it will cause deadlock.</p><pre>package main  <br>import (  <br>    &quot;fmt&quot;<br>)<br><br>var x  = 0 <br><br>func worker(chanStatus chan bool) {  <br><br>    x += 1<br><br>    chanStatus &lt;- true<br><br>}<br><br>func main() {  <br><br>    var chanStatus = make(chan bool)<br><br>    for i := 1; i &lt;= 5; i++ {       <br>	go worker(chanStatus)<br><br>	&lt;-chanStatus<br>    }<br><br><br><br>    fmt.Println(&quot;final value of x&quot;, x)<br>}</pre><p>Mutex should only be used when there are multiple threads trying to read and write from the same memory deadlock might occur. To prevent it from deadlock mutex is the best solution. Mutex locks the section from other threads to access it until it unlocks it again. There are read lock as well as write locks and even read-write locks in Golang. For race condition, thread synchronization is enough.<br>Mutex Should be used in some cases where multiple goroutines are trying to delete from hashmap, writing to a socket etc.</p><p>Now we will talk about memory management. Golang has garbage collector still when you write too much of data to a socket and receive simultaneously there is huge memory consumption and then your application might get OOM issues. To use profiling in Golang for what reason application is consuming too much of memory you can use Golang profiler.<br>Profiling is useful for identifying expensive or frequently called sections of code. The Go runtime provides <a href="https://golang.org/pkg/runtime/pprof/">profiling data</a> in the format expected by the <a href="https://github.com/google/pprof/blob/master/doc/README.md">pprof visualization tool</a>. The profiling data can be collected during testing via go test or endpoints made available from the <a href="https://golang.org/pkg/net/http/pprof/">net/http/pprof</a> package. Users need to collect the profiling data and use pprof tools to filter and visualize the top code paths.</p><p>Predefined profiles provided by the <a href="https://golang.org/pkg/runtime/pprof">runtime/pprof</a> package:</p><ul><li><strong>Cpu</strong>: CPU profile determines where a program spends its time while actively consuming CPU cycles (as opposed to while sleeping or waiting for I/O).</li><li><strong>Heap</strong>: Heap profile reports memory allocation samples; used to monitor current and historical memory usage, and to check for memory leaks.</li><li><strong>Threadcreate</strong>: Thread creation profile reports the sections of the program that lead to the creation of new OS threads.</li><li><strong>goroutine</strong>: Goroutine profile reports the stack traces of all current goroutines.</li><li><strong>Block</strong>: Block profile shows where goroutines block waiting on synchronization primitives (including timer channels). Block profile is not enabled by default; use runtime.SetBlockProfileRate to enable it.</li><li><strong>Mutex</strong>: Mutex profile reports the lock contentions. When you think your CPU is not fully utilized due to a mutex contention, use this profile. Mutex profile is not enabled by default, see runtime.SetMutexProfileFraction to enable it.</li></ul><p>If you use channels in Golang, try to use buffered channels. It uses less memory and is faster than unbuffered channels. Also, try to use <strong>“select“</strong> to avoid blocking io.</p><pre>select {<br><br>	case message, ok := &lt;-BucketData:	<br><br>		if ok{<br><br>			// some task<br>		}		<br>		break<br>	default:<br>		&lt;-time.After(1 * time.Nanosecond)<br>		break<br>}</pre><p>On more module which helps to terminate goroutines and channels. It is called context. With the help of context, you can set a timeout in go channels and terminate if the channels do not give data within the time set.</p><pre>var someChannel = make(chan bool)<br>ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutTime) * time.Millisecond)<br>defer cancel()<br><br>go someWork(someChannel)<br><br>select{<br>  case &lt;-ctx.Done():<br>    // context timeout<br>  case callback := &lt;-callbackChan:<br>    // callback from channel<br>}</pre><p>I hope this blog gives you a good understanding of channels, mutex, context, sync and goroutines.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aab81accdf83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Byte Parser]]></title>
            <link>https://sudeepdasgupta.medium.com/byte-parser-93bc7b7ee927?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/93bc7b7ee927</guid>
            <category><![CDATA[hex]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[parser]]></category>
            <category><![CDATA[binary]]></category>
            <category><![CDATA[bytebuffer]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Tue, 10 Mar 2020 17:30:55 GMT</pubDate>
            <atom:updated>2020-03-10T17:44:06.145Z</atom:updated>
            <content:encoded><![CDATA[<p>Byte Buffer to Hashmap and vice versa</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-XFnzp6Ets1xVju0MW0b8Q.jpeg" /></figure><p><strong>Exchange Parser for fast data transmission</strong></p><p>This repository is all about parsing byte array to hashmap and vice versa. Why one should use byte array rather than JSON. There are many reasons behind that</p><ol><li>Small Packet</li><li>Hackers cannot easily decode unless they have the schema</li><li>Used mainly by exchanges.</li></ol><p>To get the library and sample code visit: <a href="https://github.com/ignition123/exchange_parser"><strong>https://github.com/ignition123/exchange_parser</strong></a></p><p>There are lots of exchanges who uses FIX/FAST protocols for encoding and decoding the packets but there are also exchanges who use byte array as they schema for transmission of data. Which exchanges use byte array well they are India’s biggest stock exchange like NSE (National Stock Exchange) and BSE (Bombay Stock Exchange).</p><pre>BSE : https://www.bseindia.com/downloads1/ETI_API_Manual_1.4.8.pdf<br>NSE/CM: https://www1.nseindia.com/technology/content/nnf/TP_CM_Trimmed_NNF_PROTOCOL_4.1.pdf<br>NSE/FO: https://www1.nseindia.com/technology/content/nnf/TP_FO_Trimmed_NNF_PROTOCOL_9.25.pdf<br>NSE/CD: <a href="https://www1.nseindia.com/technology/content/nnf/TP_COM_Trimmed_NNF_PROTOCOL_1.5.pdf">https://www1.nseindia.com/technology/content/nnf/TP_COM_Trimmed_NNF_PROTOCOL_1.5.pdf</a></pre><p>Using this parser you can easily convert byte array to hashmap and hashmap to byte array.</p><pre>package main</pre><pre>import(<br>  &quot;fmt&quot;<br>    &quot;byte_parser&quot;<br>)	</pre><pre>func main(){</pre><pre>  // creating hashmap<br>  <br>  var jsonMap = make(map[string]interface{})</pre><pre>  jsonMap[&quot;BodyLen&quot;] = 176<br>  jsonMap[&quot;TemplateID&quot;] = 28500<br>  jsonMap[&quot;NetworkMsgID&quot;] = &quot;&quot;<br>  jsonMap[&quot;Pad2&quot;] = &quot;&quot;<br>  jsonMap[&quot;MsgSeqNum&quot;] = 1<br>  jsonMap[&quot;SenderSubID&quot;] = 123456<br>  jsonMap[&quot;Price&quot;] = 850000000<br>  jsonMap[&quot;StopPx&quot;] = 850000000<br>  jsonMap[&quot;MaxPricePercentage&quot;] = 850000000<br>  jsonMap[&quot;SenderLocationID&quot;] = 21212121<br>  jsonMap[&quot;ClOrdID&quot;] = 43566763434<br>  jsonMap[&quot;MessageTag&quot;] = 2222<br>  jsonMap[&quot;OrderQty&quot;] = 2<br>  jsonMap[&quot;MaxShow&quot;] = 5<br>  jsonMap[&quot;ExpireDate&quot;] = 1583693033<br>  jsonMap[&quot;MarketSegmentID&quot;] = 33232<br>  jsonMap[&quot;SimpleSecurityID&quot;] = 545454<br>  jsonMap[&quot;PartyIDTakeUpTradingFirm&quot;] = &quot;TEST1&quot;<br>  jsonMap[&quot;PartyIDOrderOriginationFirm&quot;] = &quot;TET&quot;<br>  jsonMap[&quot;PartyIDBeneficiary&quot;] = &quot;TTETST&quot;<br>  jsonMap[&quot;AccountType&quot;] = 1<br>  jsonMap[&quot;ApplSeqIndicator&quot;] = 2<br>  jsonMap[&quot;Side&quot;] = 3<br>  jsonMap[&quot;OrdType&quot;] = 2.0<br>  jsonMap[&quot;PriceValidityCheckType&quot;] = 3<br>  jsonMap[&quot;TimeInForce&quot;] = 3<br>  jsonMap[&quot;ExecInst&quot;] = 1<br>  jsonMap[&quot;TradingSessionSubID&quot;] = 2<br>  jsonMap[&quot;TradingCapacity&quot;] = 4<br>  jsonMap[&quot;Account&quot;] = &quot;IN&quot;<br>  jsonMap[&quot;PositionEffect&quot;] = &quot;Y&quot;<br>  jsonMap[&quot;RegulatoryText&quot;] = &quot;79LABS&quot;<br>  jsonMap[&quot;FreeText1&quot;] = &quot;79LABS&quot;<br>  jsonMap[&quot;FreeText2&quot;] = &quot;79LABS&quot;<br>  jsonMap[&quot;FreeText3&quot;] = &quot;79LABS&quot;<br>  jsonMap[&quot;Pad3&quot;] = &quot;LAB&quot;</pre><pre>  fmt.Println(jsonMap)</pre><pre>  // reading the schema of the byte array<br>  <br>  var byteParser = &amp;byte_parser.ByteParser{<br>    ParserType:&quot;json&quot;,<br>    Path:&quot;D:\\pounze_go_project\\crypto_backend_engine\\src\\templates\\request_templates\\json&quot;,<br>    EncodingType:&quot;bigendian&quot;,<br>  }<br>  <br>  // loading the schema<br>  <br>  byteParser.ParseJson()</pre><pre>  // converting hashmap to byte array<br>  <br>  // &quot;28500&quot; is the file name unique id to know the schema<br>  <br>  var byteArray = byteParser.ParseJsonToByte(jsonMap, &quot;28500&quot;)</pre><pre>  fmt.Println(byteArray)</pre><pre>  // converting byte array to hashmap<br>  <br>  var obj =  byteParser.ParseToObject(byteArray, &quot;28500&quot;)</pre><pre>  fmt.Println(obj)</pre><pre>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93bc7b7ee927" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang GRPC Implementation]]></title>
            <link>https://sudeepdasgupta.medium.com/golang-grpc-implementation-826adfc2543c?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/826adfc2543c</guid>
            <category><![CDATA[protocol-buffers]]></category>
            <category><![CDATA[remote-procedure-calls]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[protobuf]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Sun, 16 Feb 2020 18:58:20 GMT</pubDate>
            <atom:updated>2020-02-16T18:58:20.791Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6ufkIOzURh0WBg_q9jrSgQ.png" /></figure><p>Step by step implementation of GRPC in Golang</p><p>GRPC is basically remote method invocation or you can say remote procedural call developed by google which uses Protocol buffers as a messaging format using HTTP/2 protocol.</p><p>gRPC is a modern open-source high-performance RPC framework that can run in any environment. It can efficiently connect services in and across data centres with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in the last mile of distributed computing to connect devices, mobile applications and browsers to backend services. To know more about GRPC visit the following link.<br> <a href="https://grpc.io/">https://grpc.io/</a></p><p>Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data — think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. To know more about protocol buffers visit the google link below.<br> <a href="https://developers.google.com/protocol-buffers">https://developers.google.com/protocol-buffers</a></p><p>There are more alternate solutions like GRPC, one of them is Apache Thrift which is same as GRPC built by Facebook and now maintained by Apache Foundation and uses JSON rather than protocol buffers.<br> <a href="https://thrift.apache.org/">https://thrift.apache.org/</a></p><p>In this blog, I will demonstrate the implementation of GRPC step by step.<br>Following are the steps to download and install protocol buffers compiler. It is developed by Google which helps to create .pb.go file</p><ol><li>Download protocol buffers compiler from <a href="https://github.com/protocolbuffers/protobuf/releases">https://github.com/protocolbuffers/protobuf/releases</a></li><li>Create a proto extension file and write the following sample code</li></ol><pre>syntax = &quot;proto3&quot;;<br><br> package proto;<br><br> message Request{<br>   int64 a = 1;<br>   int64 b = 2;<br> }<br><br> message Response{<br>   int64 result = 1;<br> }<br><br> service AddService{<br>   rpc Add(Request) returns (Response);<br>   rpc Multiply(Request) returns (Response);<br> }</pre><p>3. Now to install GRPC for Golang write the following command “go get -u google.golang.org/grpc”</p><p>4. Now after download go package write the following command in the project root directory protoc.exe — proto_path=proto — proto_path=google.golang.org/grpc — go_out=plugins=grpc:proto service.proto</p><p>5. After executing the command it creates a file named “service.pb.go”. It has the schema and method and auto-generated source code.</p><p>6. Now create a main.go file in the server folder and main.go in the client folder</p><p>7. Now in the server folder main.go write the following code</p><pre>package main<br> <br> // import the following packages<br><br> import(<br>   &quot;context&quot;<br>   &quot;proto&quot;<br>   &quot;net&quot;<br>   &quot;google.golang.org/grpc&quot;<br>   &quot;google.golang.org/grpc/reflection&quot;<br> )<br> <br> // create a struct with name server<br><br> type server struct{}<br> <br> // main function<br><br> func main(){<br> <br>     // listening to tcp port 4040<br><br>     listener, err := net.Listen(&quot;tcp&quot;, &quot;:4040&quot;)<br><br>     if err != nil{<br>       panic(err)<br>     }<br>     <br>     // hosting grpc server, it is insecure server<br><br>     srv := grpc.NewServer()<br>     <br>     // Registering Add Service Method<br><br>     proto.RegisterAddServiceServer(srv, &amp;server{})<br><br>     reflection.Register(srv)<br><br>     if e:= srv.Serve(listener); e != nil{<br>       panic(err)<br>     }<br><br> }<br> <br> // creating Add method<br><br> func (s *server) Add(ctx context.Context, request *proto.Request) (*proto.Response, error){<br> <br>     // getting the value of A and B<br><br>     a, b := request.GetA(), request.GetB()<br>     <br>     // adding them and return the value to the response object of the GRPC<br><br>     result := a + b<br><br>     return &amp;proto.Response{Result: result}, nil<br><br> }<br> <br> // declaring multiply method<br><br> func (s *server) Multiply(ctx context.Context, request *proto.Request) (*proto.Response, error){<br> <br>     // getting the value of A and B<br><br>     a, b := request.GetA(), request.GetB()<br>     <br>     // multiplying a and b<br><br>     result := a * b<br>     <br>     // returning the result to the GRPC response object<br><br>     return &amp;proto.Response{Result: result}, nil<br><br> }</pre><p>8. Create the client folder main.go and write the following code</p><pre>package main<br><br>// importing the packages<br><br>import(<br>  &quot;context&quot;<br>  &quot;log&quot;<br>  &quot;time&quot;<br>  &quot;google.golang.org/grpc&quot;<br>  pb &quot;proto&quot;<br>)<br><br>// creating a constant with tcp address <br><br>const (<br>  address = &quot;localhost:4040&quot;<br>)<br><br>// declaring main method<br><br>func main(){<br><br>  // creating a connect object and connecting to tcp server<br><br>  // Set up a connection to the server.<br>  conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())<br>  <br>  if err != nil {<br>    log.Fatalf(&quot;did not connect: %v&quot;, err)<br>  }<br>  defer conn.Close()<br>  <br>  // invoking the client service to connect to grpc server<br><br>  c := pb.NewAddServiceClient(conn)<br><br>  // Contact the server and print out its response.<br>  <br>  // setting timeout for go channels<br><br>  ctx, cancel := context.WithTimeout(context.Background(), time.Second)<br><br>  defer cancel()<br>  <br>  // adding values to Add method<br><br>  r, err := c.Add(ctx, &amp;pb.Request{A: 5, B:2})<br><br>  if err != nil {<br>    log.Fatalf(&quot;could not greet: %v&quot;, err)<br>  }<br>  <br>  // getting result from server<br><br>  log.Printf(&quot;Greeting: %s&quot;, r.GetResult())<br>  <br>  // adding values to Multiply method<br><br>  r, err = c.Multiply(ctx, &amp;pb.Request{A: 5, B:6})<br><br>  if err != nil {<br>    log.Fatalf(&quot;could not greet: %v&quot;, err)<br>  }<br>  <br>  // getting result from server<br><br>  log.Printf(&quot;Greeting: %s&quot;, r.GetResult())<br>}</pre><p>There is more method to create SSL secure connections. Follow the original documentation in the following link. <a href="https://github.com/grpc/grpc-go">https://github.com/grpc/grpc-go</a>. Till then happing coding.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=826adfc2543c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Haste Golang Light Weight Web Framework]]></title>
            <link>https://sudeepdasgupta.medium.com/haste-golang-light-weight-web-framework-8b324f635f0f?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b324f635f0f</guid>
            <category><![CDATA[framework]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[middleware]]></category>
            <category><![CDATA[haste]]></category>
            <category><![CDATA[graphql]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Sat, 18 Jan 2020 09:02:10 GMT</pubDate>
            <atom:updated>2020-01-19T17:16:06.048Z</atom:updated>
            <content:encoded><![CDATA[<p>Here are the tutorials for Golang Haste Framework</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Khtrc_kz9pJminvlzwjbXg.png" /></figure><p>Before step into the framework, lets better understand what is Golang and why do we need a framework.</p><p><strong>Golang</strong> is a <a href="https://en.wikipedia.org/wiki/Static_typing">statically type</a>d, <a href="https://en.wikipedia.org/wiki/Compiled_language">compiled</a> <a href="https://en.wikipedia.org/wiki/Programming_language">programming language</a> designed at <a href="https://en.wikipedia.org/wiki/Google">Google</a> by Robert Griesemer, <a href="https://en.wikipedia.org/wiki/Rob_Pike">Rob Pike</a>, and <a href="https://en.wikipedia.org/wiki/Ken_Thompson">Ken Thompson</a>. Go is <a href="https://en.wikipedia.org/wiki/Syntax_(programming_languages)">syntactically</a> similar to <a href="https://en.wikipedia.org/wiki/C_(programming_language)">C</a>, but with <a href="https://en.wikipedia.org/wiki/Memory_safety">memory safety</a>, <a href="https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)">garbage collection</a>, <a href="https://en.wikipedia.org/wiki/Structural_type_system">structural typing</a>, and <a href="https://en.wikipedia.org/wiki/Communicating_sequential_processes">CSP</a>-style <a href="https://en.wikipedia.org/wiki/Concurrency_(computer_science)">concurrency</a>.<br>Golang has routes inbuild where in other languages you may find this in some framework like for example in Nodejs you will not find inbuild routes, but in Express framework which is built over Nodejs has routes facilities.<br>In Golang even we have template engine for rendering HTML and binding variables and loops over it, which again you won’t find in other languages directly you might have to use blade template, or jade template etc. Then if everything is inbuild in Golang why do we need a framework, well basically everyone does not have the same standard code of writing. We might need to develop middlewares and even some extra concurrency to do the task faster.</p><p>Here in Haste framework, we have inbuild middlewares, which you might need to validate session for example, in every request you receive. In this framework, we have used goroutines and even context module using channels to do the task faster and send the message from one module to other using channels. Future scope of this framework, we will have ODM for MongoDB and other NoSQL databases, ORM for SQL databases, inbuild connectivity with HDFS, MongoDB, SQL, Cassandra, Redis etc.</p><p>We have on more feature in this framework is the term we call projection. Suppose you have 5 modules in UI and need to call 5 REST APIs for that to get the message and then populate in the UI, rather than calling 5 APIs separately we will send 1 request with 5 method name and request inside the payload for example.</p><p><strong>{<br> “Login”:{ <br> “username”:”sudeep.dasgupta”,<br> “password”:”kaihiwatari”<br> },<br> “GetProfile”:{},<br> “SchemesMaster”:{<br> “category”:”EQUITY”<br> }<br>}</strong></p><p>This payload has three methods one is login, the second one is GetProfile and last is SchemeMaster now rather than calling three Apis a single request was send along with the method name and all the three response is received. Now this method a drawback, it currently only supports JSON payload along with content-type: application/json.</p><p>One more thing about this projection is that you can send a response using http/1.1 but it will send response only after all the responses are prepared in the backend. But if you use WebSocket or http/2 streams then it will send the message only when the single method response is prepared and will eventually it will push all other after their message is prepared one by one.<br>This projection is actually inspired by Google GRPC, apache thrift and GraphQL. <br>To get the complete source code visit: <a href="https://github.com/pounze/go_haste">https://github.com/pounze/go_haste</a></p><p>Bellow are the small snippet of creating routes middlewares and projection.</p><ol><li>Server.go page for initializing webserver</li></ol><pre>package main<br><br>import(<br>	&quot;cns&quot;<br>	&quot;Web&quot;<br>	&quot;net/http&quot;<br>	&quot;fmt&quot;<br>)<br><br>func main(){<br><br>	httpApp := cns.Http{}<br><br>	Web.Routes()<br><br>	go func(){<br>		defer cns.CreateHttpServer(&quot;:8000&quot;)<br>	}()<br><br>	defer cns.CreateHttpStreaming(&quot;:8100&quot;, &quot;./ssl/https-server.crt&quot;,&quot;./ssl/https-server.key&quot;, &quot;/streaming&quot;)<br>	<br>	// we can also create https server <br>	<br>	defer cns.CreateHttpsServer(&quot;:8300&quot;, &quot;./ssl/https-server.crt&quot;,&quot;./ssl/https-server.key&quot;, &quot;/&quot;)<br>	<br>	// for http/2 server we write the following code<br>	<br>	defer cns.CreateHttp2Server(&quot;:8300&quot;, &quot;./ssl/https-server.crt&quot;,&quot;./ssl/https-server.key&quot;, &quot;/&quot;)<br><br>	defer httpApp.DefaultMethod(func(req *http.Request,res http.ResponseWriter){<br>		res.Header().Set(&quot;Name&quot;, &quot;Sudeep Dasgupta&quot;)<br>		res.Header().Set(&quot;Content-Type&quot;, &quot;application/json&quot;)<br>		res.Header().Set(&quot;Access-Control-Allow-Origin&quot;, &quot;*&quot;)<br>		res.Header().Set(&quot;Access-Control-Allow-Headers&quot;, &quot;Origin, X-Requested-With, Content-Type, Accept&quot;)<br>		fmt.Println(&quot;Default header executed&quot;)<br>	})<br>}</pre><p>Here we have created 2 web servers 1 in http/1.1 and one in Http Streaming for projection. Then we have called a DefaultMethod which will be triggered whenever a new request is received. It can be used to set CORRS headers etc.</p><p>We write all routes inside the Web/RouteList.go</p><pre>import(<br>	&quot;cns&quot;<br>	&quot;fmt&quot;<br>	&quot;net/http&quot;<br>)<br><br>func Routes(){<br>	httpApp := cns.Http{}<br>	<br>	// for projection api create schema <br>	<br>	httpApp.CreateSchema(map[string]cns.Projection{<br>	    &quot;Login&quot;: cns.Projection{<br>		sample.Login,<br>		10,<br>	    },<br>	    &quot;GetProfile&quot;: cns.Projection{<br>		sample.GetProfile,<br>		10,<br>	    },<br>	});<br>	<br>	// set route path for projection schema add middleware method to it<br>	<br>	httpApp.SetRoutePath(&quot;/&quot;).Middlewares(func(req *http.Request,res http.ResponseWriter,done chan bool){<br>		fmt.Println(&quot;middleware worked&quot;)<br>		done &lt;- true<br>	})<br>	<br>	// projection api for socket support <br>	<br>	httpApp.SetSocketRoutePath(&quot;/realtime&quot;).Middlewares(func(req *http.Request,res http.ResponseWriter,done chan bool){<br>		fmt.Println(&quot;middleware worked&quot;)<br>		done &lt;- true<br>	})<br>	<br>	// block directories to get access<br>	<br>	httpApp.BlockDirectories([]string{&quot;/UserView/img/&quot;,&quot;/UserView/js/&quot;})<br><br>	hm := map[string]string{<br>	    &quot;$id&quot;:&quot;[0-9]{2}&quot;,<br>	    &quot;$name&quot;:&quot;[a-z]+&quot;,<br>	}<br>	<br>	// to save file in case of multipart form data<br>	<br>	httpApp.SaveFile(w http.ResponseWriter, file multipart.File, path string, fileChan chan bool)<br><br>	// url matching using regular expression and calling multiple middleware with chaining<br>	<br>	httpApp.Post(&quot;/$id/$name[\\/]*&quot;,func(req *http.Request,res http.ResponseWriter){<br>		fmt.Println(&quot;method invoked&quot;)<br>		fmt.Println(req.URL.Query().Get(&quot;$name&quot;))<br>		fmt.Fprintf(res, &quot;Successfully Done&quot;)<br>	}).Middlewares(func(req *http.Request,res http.ResponseWriter,done chan bool){<br><br>		err := req.ParseMultipartForm(200000)<br>	if err != nil {<br>	    fmt.Println(&quot;Unable to parse form data&quot;)<br>	    return<br>	}<br>	_,handle,_ := req.FormFile(&quot;name&quot;)<br>	fmt.Println(handle)<br><br>		done &lt;- true<br>	}).Middlewares(func(req *http.Request,res http.ResponseWriter,done chan bool){<br>		fmt.Println(&quot;working both&quot;)<br>		done &lt;- true<br>	}).Where(hm)<br><br>	// calling Get request method same POST, PUT and DELETE available<br>	<br>	httpApp.Get(&quot;/&quot;,func(req *http.Request,res http.ResponseWriter){<br>		//cns.Push(res,&quot;/UserView/js/test.js&quot;)<br>		stat,_,result := cns.Authorization(req,res,&quot;Enter username and password to Authenticate&quot;)<br>		if stat{<br>			fmt.Println(result)<br>			http.ServeFile(res, req, &quot;src/views/index.html&quot;)<br>		}else{<br>			fmt.Fprintf(res, &quot;Unauthorized&quot;)<br>		}<br>	})<br>	<br>	// TO call global middleware<br>	<br>	httpApp.GlobalMiddleWares(func(req *http.Request,res http.ResponseWriter,done chan bool){<br>		fmt.Println(&quot;working both&quot;)<br>		done &lt;- true<br>	})<br><br>	// creating try catch block to handle exception<br>	<br>	cns.Block{<br>		Try:func(){<br>			fmt.Println(&quot;I tried&quot;)<br>			cns.Throw(&quot;ohhh&quot;)<br>			fmt.Println(&quot;Working&quot;)<br>		},<br>		Catch:func(e cns.Exception){<br>			fmt.Println(&quot;caught exception&quot;,e)<br>		},<br>		Finally:func(){<br>			fmt.Println(&quot;Finally&quot;)<br>		},<br>	}.Do()<br>}</pre><p>For the Config.go, it is inside the cns/Config.go<br>There are lots of framework like gin, buffalo but you can benchmark this Haste with them and you may love it as its simple, fast and lightweight. We will try to improve the documentation better. Till then happy coding.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b324f635f0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RPC With Nodejs]]></title>
            <link>https://sudeepdasgupta.medium.com/rpc-with-nodejs-de934668ce5a?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/de934668ce5a</guid>
            <category><![CDATA[http-request]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[rpc]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[http2]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Tue, 10 Dec 2019 19:02:46 GMT</pubDate>
            <atom:updated>2020-06-12T18:49:28.508Z</atom:updated>
            <content:encoded><![CDATA[<h3>RPC With NodeJS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-p1mIBjmtKlglRAzqPrHw.png" /></figure><h3>RPC with Nodejs Websocket and http/2 Push Streaming, inspired from apache thrift, Grpc</h3><p>This is a framework built over Nodejs express framework and ws library. It is built as inspiration google GRPC and Apache Thrift. In this, you can invoke methods using http/1.1 and get streaming packets using http/2 push and WebSockets.</p><p>To get complete source code and example visit the following url.<br><a href="https://github.com/ignition123/node_projection_api"><strong>https://github.com/ignition123/node_projection_api</strong></a></p><p>// All request are post request, run the server.js it will establish http/1.1, WebSocket and http/2 push server</p><p>// Make post request with the following payload and header as Content-Type: <strong>application/json</strong> , send this payload with <strong>http/2</strong> client code sample written at the bottom<br>// All examples for client request and written at the bottom<br>// Make websocket message payload with the following<br>// For http/2 stream packets are splitted with \r\n\r\n</p><pre>{<br>	&quot;Login&quot;:{<br>		&quot;requestPayload&quot;:{<br>			&quot;username&quot;:&quot;Joe&quot;,<br>			&quot;password&quot;:&quot;kaihiwatari&quot;<br>		}<br>	},<br>	&quot;GetProfile&quot;:{},<br>	&quot;SchemeMaster&quot;:{<br>		&quot;requestPayload&quot;:{<br>			&quot;category&quot;:&quot;EQUITY&quot;<br>		}<br>	}<br>}</pre><p>// Here we are importing Project parser module</p><pre>const parserSchema = require(&quot;./ProjectParser&quot;);</pre><p>// This are routes we have created 2 routes inside controllers folder one is Login and the other is GetProfile</p><pre>const Login = require(&quot;./controllers/Login&quot;);</pre><pre>const GetProfile = require(&quot;./controllers/GetProfile&quot;);</pre><p>// Here we are creating schema for method invocation. // It requires 2 parameters<br>// Method Object // Timeout for the request to complete, if the routes is not able to give response in between the timeout then it will terminate the route and give timeout response to client</p><pre>parserSchema.Project.CreateSchema({<br>    &quot;Login&quot;:{<br>	method:Login,<br>	timeout:100,<br>    },<br>    &quot;GetProfile&quot;:{<br>	method:GetProfile,<br>	timeout:100,<br>    }<br>});</pre><p>// Here we are invoking express framework use method to create interceptor</p><pre>parserSchema.Project.Use(function(req, res, next)<br>{<br>    res.header(&quot;Access-Control-Allow-Origin&quot;, &quot;*&quot;);<br>    res.header(&quot;Access-Control-Allow-Headers&quot;, &quot;Origin, X-Requested-With, Content-Type, Accept&quot;);<br>    next();<br>});</pre><p>// Here we are starting Http/1.1 listener with http module, you can pass https also for https you can call <br>// parserSchema.Project.Listen(“https”,”127.0.0.1:8000&quot;, sslOptions);</p><pre>parserSchema.Project.Listen(&quot;http&quot;,&quot;127.0.0.1:8000&quot;);</pre><p>// Here are setting path for remote method invocation</p><pre>parserSchema.Project.SetRoutePath(&quot;/&quot;);</pre><p>// Here we are starting websocket streaming for remote method invocation, it will listen to the same port as http server</p><pre>parserSchema.Project.StartStreaming();</pre><p>// here we are create http2 streaming for push</p><pre>parserSchema.Project.CreateHttpStreaming(&quot;http&quot;,&quot;127.0.0.1:8100&quot;);</pre><p>// here we are starting the streaming</p><pre>parserSchema.Project.StartHttpStreaming();</pre><p>##############################################################################################################################</p><p>// For client setup for http/1.1 and websocket</p><pre>&lt;html&gt;</pre><pre>    &lt;script<br>	src=&quot;https://code.jquery.com/jquery-2.2.4.min.js&quot;<br>	integrity=&quot;sha256-BbhdlvQf/xTY9gja0Dq3HiwQF8LaCRTXxZKRutelT44=&quot;<br>	crossorigin=&quot;anonymous&quot;&gt;&lt;/script&gt;<br>    &lt;script&gt;</pre><pre>	var saveData = $.ajax({<br>	    type: &#39;POST&#39;,<br>	    url: &quot;http://localhost:8000&quot;,<br>	    beforeSend: function(request) {<br>		request.setRequestHeader(&quot;Content-Type&quot;, &quot;application/json&quot;);<br>	    },<br>	    data: JSON.stringify({<br>		&quot;Login&quot;:{<br>		    &quot;requestPayload&quot;:{<br>			&quot;username&quot;:&quot;sudeep.dasgupta&quot;,<br>			&quot;password&quot;:&quot;kaihiwatari&quot;<br>		    }<br>		},<br>		&quot;GetProfile&quot;:{},<br>		&quot;SchemesMaster&quot;:{<br>		    &quot;requestPayload&quot;:{<br>			&quot;category&quot;:&quot;EQUITY&quot;<br>		    }<br>		}<br>	    }),<br>	    success: function(resultData)<br>	    { <br>		console.log(resultData)<br>	    }<br>	});<br>	function WebSocketTest()<br>	{<br>	    if (&quot;WebSocket&quot; in window)<br>	    {               <br>	       // Let us open a web socket<br>	       var ws = new WebSocket(&quot;ws://localhost:8000&quot;);</pre><pre>	       ws.onopen = function()<br>	       {</pre><pre>		  // Web Socket is connected, send data using send()<br>		  ws.send(<br>		    JSON.stringify({<br>			&quot;Login&quot;:{<br>			    &quot;requestPayload&quot;:{<br>				&quot;username&quot;:&quot;sudeep.dasgupta&quot;,<br>				&quot;password&quot;:&quot;kaihiwatari&quot;<br>			    }<br>			},<br>			&quot;GetProfile&quot;:{},<br>			&quot;SchemesMaster&quot;:{<br>			    &quot;requestPayload&quot;:{<br>				&quot;category&quot;:&quot;EQUITY&quot;<br>			    }<br>			}<br>		    })<br>		  );<br>	       };</pre><pre>	       ws.onmessage = function (evt)<br>	       { <br>		  var received_msg = evt.data;<br>		  console.log(received_msg);<br>	       };</pre><pre>	       ws.onclose = function()<br>	       { </pre><pre>		  // websocket is closed.<br>		  console.log(&quot;Connection is closed...&quot;); <br>	       };<br>	    } else {</pre><pre>	       // The browser doesn&#39;t support WebSocket<br>	       console.log(&quot;WebSocket NOT supported by your Browser!&quot;);<br>	    }<br>	 }<br>	 WebSocketTest();<br>    &lt;/script&gt;<br>&lt;/html&gt;</pre><p>##################################################################################################################</p><p>// For client http/2 push streams</p><pre>const http2 = require(&#39;http2&#39;);<br>function post(url, path, body)<br>{</pre><pre>    const client = http2.connect(url);<br>    const buffer = Buffer.from(JSON.stringify(body));<br>    const req = client.request({<br>	[http2.constants.HTTP2_HEADER_SCHEME]: &quot;http&quot;,<br>	[http2.constants.HTTP2_HEADER_METHOD]: http2.constants.HTTP2_METHOD_POST,<br>	[http2.constants.HTTP2_HEADER_PATH]: `/${path}`,<br>	&quot;Content-Type&quot;: &quot;application/json&quot;,<br>	&quot;Content-Length&quot;: buffer.length,<br>    });<br>    req.on(&#39;response&#39;, (headers, flags) =&gt; {<br>	for (const name in headers) {<br>	  console.log(`${name}: ${headers[name]}`);<br>	}<br>    });<br>    req.setEncoding(&#39;utf8&#39;);<br>    var data = &quot;&quot;;<br>    req.on(&#39;data&#39;, (chunk) =&gt; {<br>	console.log(chunk);<br>	data += chunk;<br>    });<br>    req.end(buffer);<br>    req.on(&#39;end&#39;, () =&gt; {<br>       console.log(data);<br>       client.close();<br>    });<br>    req.on(&quot;error&quot;,function(err)<br>    {<br>	console.log(err);<br>    })<br>}</pre><pre>post(&quot;http://127.0.0.1:8100&quot;, &quot;/&quot;, {<br>	&quot;Login&quot;:{<br>		&quot;requestPayload&quot;:{<br>			&quot;username&quot;:&quot;sudeep.dasgupta&quot;,<br>			&quot;password&quot;:&quot;kaihiwatari&quot;<br>		}<br>	},<br>	&quot;GetProfile&quot;:{},<br>	&quot;SchemesMaster&quot;:{<br>		&quot;requestPayload&quot;:{<br>			&quot;category&quot;:&quot;EQUITY&quot;<br>		}<br>	}<br>});</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de934668ce5a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Byte Buffer with Golang]]></title>
            <link>https://sudeepdasgupta.medium.com/byte-buffer-with-golang-a6c0f6d4e470?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6c0f6d4e470</guid>
            <category><![CDATA[bytebuffer]]></category>
            <category><![CDATA[stream]]></category>
            <category><![CDATA[bytearray]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[binary]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Sat, 07 Dec 2019 07:28:40 GMT</pubDate>
            <atom:updated>2019-12-14T08:11:04.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*rn2y3Dma399-4iZD8jTOAw.png" /></figure><p>Byte Buffer with Golang</p><p>Golang ByteBuffer Library</p><p>This library helps you to work with bytes packets and create a buffer array very easily in Golang. Here you can put, get bytes with its data types.</p><p>To get the package and source code go to the following link: <a href="https://github.com/pounze/ByteBuffer_golang"><strong>https://github.com/pounze/ByteBuffer_golang</strong></a></p><p>There are several methods to do it. Followign are the examples of ByteBufferPacket</p><p>To import package you can do the following<br><strong>import ( “fmt” “ByteBuffer” )</strong></p><p>Now we will run the method one by one<br>To Initialize the ByteBuffer packet we do the following.<br><strong>var byteBuffer = ByteBuffer.Buffer{}</strong></p><p>To put short value that is uint16 in golang<br><strong>byteBuffer.PutShort(20)</strong></p><p>To put int value that is uint32 in golang<br><strong>byteBuffer.PutInt(20)</strong></p><p>To put long value that is uint64 in golang<br><strong>byteBuffer.PutLong(20)</strong></p><p>To put float value that is float32 in golang<br><strong>byteBuffer.PutFloat(20.0)</strong></p><p>To put double value that is float64 in golang<br><strong>byteBuffer.PutDouble(20.0)</strong></p><p>To put String value<br><strong>var sampleData = []byte(“Hello World”) byteBuffer.Put(sampleData)</strong></p><p>To convert ByteBuffer to byte array<br><strong>var res = byteBuffer.Array()</strong></p><p>To convert byte array to ByteBuffer<br><strong>var byteBufferNew = ByteBuffer.Buffer{} byteBufferNew.Wrap(res)</strong></p><p>To fetch short value from byte buffer<br><strong>var shortData = byteBufferNew.GetShort()<br>fmt.Println(shortData)</strong></p><p>To fetch int value from byte buffer<br><strong>var intData = byteBufferNew.GetInt()<br>fmt.Println(intData)</strong></p><p>To fetch double value from byte buffer<br><strong>var longData = byteBufferNew.GetLong()<br>fmt.Println(longData)</strong></p><p>To fetch float value from byte buffer<br><strong>var floatData = byteBufferNew.GetFloat()<br>fmt.Println(floatData)</strong></p><p>To fetch double value from byte buffer<br><strong>var doubleData = byteBufferNew.GetDouble()<br>fmt.Println(doubleData)</strong></p><p>To fetch bytes value from byte buffer<br><strong>var data = byteBufferNew.Get(len(sampleData))<br>fmt.Println(data)</strong></p><p>To get size value from byte buffer<br><strong>fmt.Println(byteBufferNew.Size())</strong></p><p>To flip byte buffer for littleEndian<br><strong>byteBuffer.Flip()<br>fmt.Println(byteBuffer.Array())</strong></p><p>To clear byte buffer<br><strong>byteBuffer.Clear()<br>fmt.Println(byteBuffer.Array())</strong></p><p>To slice a bytebuffer<br><strong>err := byteBuffer.Slice(2,2)<br>if err != nil{ fmt.Println(err) }</strong></p><p>To convert byte array to string<br><strong>fmt.Println(byteBuffer.Bytes2Str(data))</strong></p><p>To convert string to byte array<br><strong>fmt.Println(byteBuffer.Str2Bytes(byteBuffer.Bytes2Str(data)))</strong></p><p>There are more method like<strong> Bytes2Short, Short2Bytes, Byte2Int, Int2Bytes, Bytes2Long, Long2Bytes, Float2Bytes, Bytes2Float, Double2Bytes, Bytes2Double</strong></p><p>More methods will be introduced very soon. Happy coding</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6c0f6d4e470" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Video Streaming with RTMP Module]]></title>
            <link>https://sudeepdasgupta.medium.com/video-streaming-with-rtmp-module-f46dea0829fe?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/f46dea0829fe</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[ffmpeg]]></category>
            <category><![CDATA[mpeg-dash]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[hls]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Sat, 30 Nov 2019 23:47:49 GMT</pubDate>
            <atom:updated>2020-02-02T10:58:35.061Z</atom:updated>
            <content:encoded><![CDATA[<h3>Video Streaming with Nginx RTMP / HLS</h3><p>Video streaming with RTMP module using Nginx webserver</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R_X9Vj2EAIGx_LEEE7mGxw.jpeg" /></figure><p>Nowadays video streaming is a trend and people are crazy about facebook live, twitch tv and some other applications, but how these applications are built, what sort of protocols are used to do such streaming and people are viewing these streams in mobile apps and browser. Well, this is the perfect blog for you to learn the complete setup of the streaming server and also some alternative solutions will be provided.<br>So, what is RTMP? According to Wikipedia,<br><strong>Real-Time Messaging Protocol</strong> (<strong>RTMP</strong>) was initially a <a href="https://en.wikipedia.org/wiki/Proprietary_protocol">proprietary protocol</a> developed by <a href="https://en.wikipedia.org/wiki/Macromedia">Macromedia</a> for <a href="https://en.wikipedia.org/wiki/Streaming_media">streaming</a> audio, video and data over the Internet, between a <a href="https://en.wikipedia.org/wiki/Adobe_Flash">Flash</a> player and a server. Macromedia is now owned by <a href="https://en.wikipedia.org/wiki/Adobe_Systems">Adobe</a>, which has released an incomplete version of the specification of the protocol for public use. <strong><em>The above image is a google cloud platform architecture taken from the website where they have shown how vod(video on demand) and live streaming works.<br></em></strong><em>To get the full source go to my git repo:</em><strong><em> </em></strong><a href="https://github.com/ignition123/NgINX-Config-RTMP"><strong>https://github.com/ignition123/NgINX-Config-RTMP</strong></a></p><p>The RTMP protocol has multiple variations:</p><ol><li>The “plain” protocol which works on top of and uses <strong>TCP port number 1935</strong> by default.</li><li><strong>RTMPS</strong>, which is RTMP over a <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS/SSL</a> connection.</li><li><strong>RTMPE</strong>, which is RTMP encrypted using Adobe’s own security mechanism. While the details of the implementation are proprietary, the mechanism uses industry-standard cryptographic primitives.<a href="https://en.wikipedia.org/wiki/Real-Time_Messaging_Protocol#cite_note-RTMPE_overview-1">[1]</a></li><li><strong>RTMPT</strong>, which is <a href="https://en.wikipedia.org/wiki/Encapsulation_(networking)">encapsulated</a> within <a href="https://en.wikipedia.org/wiki/HTTP">HTTP</a> requests to traverse firewalls. RTMPT is frequently found utilizing cleartext requests on <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a> <a href="https://en.wikipedia.org/wiki/Port_(computer_networking)">ports</a> 80 and 443 to bypass most corporate traffic filtering. The encapsulated session may carry plain RTMP, RTMPS, or RTMPE packets within.</li><li><strong>RTMFP</strong>, which is RTMP over <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a> instead of TCP, replacing RTMP Chunk Stream. The Secure <a href="https://en.wikipedia.org/wiki/Real-Time_Media_Flow_Protocol">Real-Time Media Flow Protocol</a> suite has been developed by Adobe Systems and enables end‐users to connect and communicate directly with each other (P2P).</li></ol><p>Facebook, twitch all use RTMPS now as it uses SSL for message transfer. Some other protocols are also there for such streaming such as RTSP which is widely used by youtube.</p><p>For the RTMP server setup there are tons of service some of them are open source and some of them are paid. Following are the services that provides RTMP support:</p><ol><li><strong>Wowza Streaming Engine</strong> (known as <strong><em>Wowza Media Server</em></strong> prior to version 4) is a unified streaming media server software developed by Wowza Media Systems. The server is used for <a href="https://en.wikipedia.org/wiki/Streaming_media">streaming</a> of live and <a href="https://en.wikipedia.org/wiki/Video_on_demand">on-demand video</a>, audio, and <a href="https://en.wikipedia.org/wiki/Rich_Internet_application">rich Internet applications</a> over <a href="https://en.wikipedia.org/wiki/IP_networks">IP networks</a> to desktop, laptop, and tablet computers, <a href="https://en.wikipedia.org/wiki/Mobile_device">mobile devices</a>, IPTV set-top boxes, internet-connected TV sets, game consoles, and other network-connected devices. The server is a <a href="https://en.wikipedia.org/wiki/Java_(programming_language)">Java</a> application deployable on most <a href="https://en.wikipedia.org/wiki/Operating_systems">operating systems</a>. The official site is <a href="https://www.wowza.com/"><strong>https://www.wowza.com/</strong></a></li><li><strong>Ant Media Server</strong> is another streaming server that supports RTMP and provides both opensource and enterprise versions. The official site is <a href="https://antmedia.io/"><strong>https://antmedia.io/</strong></a></li><li><strong>Red 5 </strong>is another streaming server just like ant media server which is built in java and provides opensource and enterprise version of it. The official site is <a href="https://www.red5pro.com/open-source/"><strong>https://www.red5pro.com/open-source/</strong></a></li><li>For some opensource libraries, there are solutions like node <strong>media server( </strong><a href="https://www.npmjs.com/package/node-media-server"><strong>https://www.npmjs.com/package/node-media-server</strong></a><strong>), golang RTMP module( </strong><a href="https://godoc.org/github.com/TrevorSStone/gortmp"><strong>https://godoc.org/github.com/TrevorSStone/gortmp</strong></a><strong>)</strong></li><li><strong>Nginx Web Server</strong>, it is a web server that provides load balancing, socket streaming, reverse proxy server and many more it comes in 2 versions, 1 is opensource and one with enterprise version. It is recently acquired by F5. <strong>( </strong><a href="https://www.nginx.com/"><strong>https://www.nginx.com/</strong></a><strong>)</strong></li></ol><p>In this blog, I will be demonstration step by step from installation to configurations of RTMP and then streaming the video in the browser or any mobile/tablet platform.</p><p>Following are the steps,</p><ol><li>You need Linux servers as the Nginx RTMP module works only in the Linux server.</li><li>You need edge CDN servers to stream in a different place with low latency. peer 5 is a good CDN server.</li><li>For RTMP to HLS or MPEG-DASH you need a c++ library called FFmpeg which is an open-source video encoding library, it is the best encoder in opensource and is very powerful to encode the video streams to any format. We will explain about HLS and MPEG -DASH in bellow.</li><li>To install Nginx you can check in google. Install Nginx version 1.6 for RTMP support. To install Nginx RTMP module use the following commands<br><strong>Download &amp; unpack latest Nginx-RTMP (you can also use HTTP)</strong></li></ol><pre><strong>sudo git clone git://github.com/arut/nginx-rtmp-module.git</strong></pre><pre>Download &amp; unpack Nginx (you can also use svn)<br><strong>sudo wget http://nginx.org/download/nginx-1.14.1.tar.gz<br>sudo tar xzf nginx-1.14.1.tar.gz<br>cd nginx-1.14.1</strong></pre><pre>Build Nginx with Nginx-RTMP<br><strong>sudo ./configure --with-http_ssl_module --add-module=../nginx-rtmp-module<br>sudo make<br>sudo make install<br>Start Nginx Server</strong></pre><pre><strong>sudo /usr/local/nginx/sbin/nginx</strong></pre><p>Now, we will change the Nginx conf file to make changes for RTMP. <br>Here we have set the worker process to auto so that Nginx scales automatically. Setting the number of filehandle for connections and then setting error log path.<strong><br>worker_processes auto; <br>worker_rlimit_nofile 100000;<br>error_log /var/log/nginx/error.log crit;</strong></p><p>Now we set up the connections configurations of nginx.</p><p><strong>events { </strong><br># determines how much clients will be served per worker <br># max clients = worker_connections * worker_processes <br># max clients is also limited by the number of socket connections available on the system (~64k)<br><strong>worker_connections 4000; </strong><br># optimized to serve many clients with each thread, essential for linux — for testing environment use <br><strong>epoll; </strong><br># accept as many connections as possible, may flood worker connections if set too low — for testing environment <br><strong>multi_accept on;</strong><br><strong>}<br></strong>Now we set up the RTMP module configuration. This uses some events like onplay, onpublish, ondone where authentication is used to verify the client. We will share the auth sample code written in nodejs later. The streaming is then converted to a hls file which is then streamed using http protocol.<br><strong>rtmp {<br>server {<br>listen 1935;</strong> # Listen on standard RTMP port<br><strong>ping 30s;<br>notify_method get;<br>chunk_size 4096;</strong><br># video on demand for flv files<br><strong>application vod {<br>play /mnt/flvs;<br>}</strong><br># video on demand for mp4 files<br><strong>application vod2 {<br>play /mnt/mp4s;<br>}</strong><br># This application is to accept incoming stream<br><strong>application live {<br>live on; # Allows live<br>allow play all;<br>on_publish </strong><a href="http://127.0.0.1:3000/rtmp_key_auth;"><strong>http://127.0.0.1:3000/rtmp_key_auth;</strong></a><strong><br>on_done </strong><a href="http://127.0.0.1:3000/rtmp_streams_end;"><strong>http://127.0.0.1:3000/rtmp_streams_end;</strong></a><br># on play event triggers when any client is streaming and displaying the streams<br>#on_play <a href="http://127.0.0.1:3000/rtmp_key_auth;">http://127.0.0.1:3000/rtmp_key_auth;</a><br># Once receive stream, transcode for adaptive streaming<br># This single ffmpeg command takes the input and transforms<br># the source into 4 different streams with different bitrate<br># and quality. P.S. The scaling done here respects the aspect<br># ratio of the input.<br><strong>exec bash /opt/live_stream/exec_wrapper.sh $app $name;<br>exec_kill_signal term;<br>record all;<br>record_path /mnt/recordings;<br>record_unique on;<br>record_append on;<br>exec_record_done ffmpeg -i /mnt/recordings/ -f mp4 /mnt/mp4s/$basename.mp4;<br>}</strong><br><strong>application live_mobile {<br>live on;<br>allow play all;</strong><br>#-max_muxing_queue_size 4000<br>#-hls_list_size 100<br><strong>on_publish </strong><a href="http://127.0.0.1:3000/rtmp_key_auth;"><strong>http://127.0.0.1:3000/rtmp_key_auth;</strong></a><strong><br>exec_push ffmpeg -re -i rtmp://127.0.0.1/$app/$name -async 1 -vsync -1 -c:v copy -c:a copy -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 10 -hls_playlist_type event /mnt/hls/$name.m3u8;<br>record all;<br>record_path /mnt/recordings;<br>record_unique on;<br>record_append on;<br>exec_record_done ffmpeg -i /mnt/recordings/ -f mp4 /mnt/mp4s/$basename.mp4;<br>}</strong><br># This application is for splitting the stream into HLS fragments<br><strong>application show {<br>live on;</strong> # Allows live input from above<br><strong>hls on;</strong> # Enable HTTP Live Streaming<br><strong>hls_fragment 3;<br>hls_playlist_length 60;</strong><br>#hls_sync 100ms;<br><strong>hls_continuous on;<br>deny play all;</strong><br># Pointing this to an SSD is better as this involves lots of IO<br><strong>hls_path /mnt/hls/;<br>hls_cleanup on;<br>record all;<br>record_path /mnt/recordings;<br>record_unique on;<br>record_append on;<br>exec_record_done ffmpeg -i /mnt/recordings/ -f mp4 /mnt/mp4s/$basename.mp4;</strong><br># Instruct clients to adjust resolution according to bandwidth<br><strong>hls_variant _low BANDWIDTH=288000;</strong> # Low bitrate, sub-SD resolution<br><strong>hls_variant _mid BANDWIDTH=448000;</strong> # Medium bitrate, SD resolution<br><strong>hls_variant _high BANDWIDTH=1152000; </strong># High bitrate, higher-than-SD resolution<br><strong>hls_variant _hd720 BANDWIDTH=2048000;</strong> # High bitrate, HD 720p resolution<br><strong>hls_variant _src BANDWIDTH=4096000; </strong># Source bitrate, source resolution<br><strong>}</strong><br><strong>application recorder {<br>live on;<br>recorder all {<br>record all;</strong><br><strong>record_path /mnt/recordings;</strong><br>#record_max_size 100000K;<br>#record_max_frames 4;<br><strong>record_unique on;</strong><br>#record_suffix _%d%m%Y_%H%M%S.flv;<br>#record_append on;<br>#record_interval 5s;<br>#record_notify on;<br><strong>exec_record_done ffmpeg -i $path -f mp4 /tmp/live/$basename.mp4;<br>}}<br>application show {</strong><br><strong>live on;</strong> # Allows live input from above<br><strong>hls on; </strong># Enable HTTP Live Streaming<br><strong>hls_fragment 3;<br>hls_playlist_length 60;</strong><br>#hls_sync 100ms;<br><strong>hls_continuous on;<br>deny play all;</strong><br># Pointing this to an SSD is better as this involves lots of IO<br><strong>hls_path /mnt/hls/;<br>hls_cleanup on;<br>record all;<br>record_path /mnt/recordings;<br>record_unique on;<br>record_append on;<br>exec_record_done ffmpeg -i /mnt/recordings/ -f mp4 /mnt/mp4s/$basename.mp4;</strong><br># Instruct clients to adjust resolution according to bandwidth<br><strong>hls_variant _low BANDWIDTH=288000;</strong> # Low bitrate, sub-SD resolution<br><strong>hls_variant _mid BANDWIDTH=448000;</strong> # Medium bitrate, SD resolution<br><strong>hls_variant _high BANDWIDTH=1152000; </strong># High bitrate, higher-than-SD resolution<br><strong>hls_variant _hd720 BANDWIDTH=2048000;</strong> # High bitrate, HD 720p resolution<br><strong>hls_variant _src BANDWIDTH=4096000;</strong> # Source bitrate, source resolution<br><strong>}</strong><br><strong>application recorder {<br>live on;<br>recorder all {<br>record all;<br>record_path /mnt/recordings;</strong><br>#record_max_size 100000K;<br>#record_max_frames 4;<br><strong>record_unique on;</strong><br>#record_suffix _%d%m%Y_%H%M%S.flv;<br>#record_append on;<br>#record_interval 5s;<br>#record_notify on;<br><strong>exec_record_done ffmpeg -i $path -f mp4 /tmp/live/$basename.mp4;<br>}}}}</strong></p><p>Now to serve the HLS streams we configure the HTTP server in Nginx. Following are the configuration</p><p><strong>http {<br>open_file_cache max=200000 inactive=20s;<br>open_file_cache_valid 30s;<br>open_file_cache_min_uses 2;<br>open_file_cache_errors on;<br>gzip on;</strong><br># gzip_static on;<br><strong>gzip_min_length 10240;<br>gzip_comp_level 1;<br>gzip_vary on;<br>gzip_disable msie6;<br>gzip_proxied expired no-cache no-store private auth;<br>gzip_types</strong><br># text/html is always compressed by HttpGzipModule<br><strong>text/css<br>text/javascript<br>text/xml<br>text/plain<br>text/x-component<br>application/javascript<br>application/x-javascript<br>application/json<br>application/xml<br>application/rss+xml<br>application/atom+xml<br>font/truetype<br>font/opentype<br>application/vnd.ms-fontobject<br>image/svg+xml;</strong><br>#access_log logs/access.log main;<br><strong>directio 512;<br>server {<br>listen 9200;<br>server_name localhost;</strong><br>#charset koi8-r;<br>#access_log logs/host.access.log main;<br><strong>location / {</strong><br># Disable cache<br>add_header ‘Cache-Control’ ‘no-cache’;<br># CORS setup<br><strong>add_header ‘Access-Control-Allow-Origin’ ‘*’ always;<br>add_header ‘Access-Control-Expose-Headers’ ‘Content-Length’;</strong><br># allow CORS preflight requests<br><strong>if ($request_method = ‘OPTIONS’) {add_header ‘Access-Control-Allow-Origin’ ‘*’;<br>add_header ‘Access-Control-Max-Age’ 1728000;<br>add_header ‘Content-Type’ ‘text/plain charset=UTF-8’;<br>add_header ‘Content-Length’ 0;<br>return 204;<br>}<br>types {<br>application/dash+xml mpd;<br>application/vnd.apple.mpegurl m3u8;<br>video/mp2t ts;<br>video/x-flv flv;<br>video/mp4 mp4;<br>video/webm webm;<br>}</strong><br><strong>root /mnt/;<br>}}}</strong></p><p>Now we are done with the configuration of nginx server for both RTMP and HLS streaming to read about HLS streaming following is the path <a href="https://en.wikipedia.org/wiki/HTTP_Live_Streaming"><strong>https://en.wikipedia.org/wiki/HTTP_Live_Streaming</strong></a>, to read about MPEG-DASH following is the path <a href="https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP"><strong>https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP</strong></a>. There are more alternative like micrtosoft smooth streaming but we dont want to cover that for now.<br>Now we will demonstrate how to authenticate client before streaming starts or ends or play,<br>Following is the NodeJS sample snippet which you can modify use database to validate whatever you like. Following is the code written in express where it matches the key with the myStream string and validates you can uses any key that you will generate and pass it to client before starting the stream.<br><strong>app.get(‘/rtmp_key_auth’,async function(req,res){<br> if(req.query.name === undefined){<br> res.status(404);<br> res.send(‘Key does not match’);<br> }else{<br> if(req.query.name === “myStream”){<br> res.status(200);<br> res.send(‘Authenticated’);<br> } else{<br> res.status(404);<br> res.send(‘Key does not match’);<br> }}});</strong></p><p>Now remember video streaming does not take too much of CPU utilization but video encoding takes a lot of CPU usage for that you need good servers, secondly if some streams in a low-resolution don&#39;t upgrade the pixels it will take lots of CPU usage, always downgrade you stream from upper to lower for HLS to dynamically change the resolution depending on the internet speed. Suppose if someone is streaming at a resolution of 1080p then we will download streams to 720p, 480p, and 360p plus the original stream we are receiving. For that, I have created a bash script that works perfectly for downgrading the streams to different resolutions for HLS to stream. NGINX will execute the bash file which will future call FFmpeg library to encode the video from RTMP streams to HLS file. Following is the bash script sample.<br> <strong>#!/bin/bash <br>on_die (){ <br># kill all children <br>pkill -KILL -P $$<br>} <br>trap ‘on_die’ TERM <br>filepath=”rtmp://172.104.55.76/$1/$2&quot;<br>width_prefix=’streams_stream_0_width=’<br>height_prefix=’streams_stream_0_height=’<br>declare -a dimensions<br>while read -r line<br>do <br>dimensions+=( “${line}” )<br>done &lt; &lt;( ffprobe -v error -of flat=s=_ -select_streams v:0 -show_entries stream=width,height “${filepath}” )<br>width_with_prefix=${dimensions[0]}<br>height_with_prefix=${dimensions[1]}<br>width=${width_with_prefix#${width_prefix}}<br>height=${height_with_prefix#${height_prefix}}<br> if [ “${height}” -lt 1441 ] &amp;&amp; [ “${height}” -gt 1080 ]<br>then <br>ffmpeg -i rtmp://172.104.55.76/$1/$2 -async 1 -vsync -1 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 400k -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:360” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_360p.m3u8 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 500K -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:480” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_480p.m3u8 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 1500K -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:720” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_720p.m3u8 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 3000K -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:1080” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_1080p.m3u8 -c copy -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2.m3u8 &amp;<br>elif [ “${height}” -lt 1081 ] &amp;&amp; [ “${height}” -gt 720 ]<br>then <br>ffmpeg -i rtmp://172.104.55.76/$1/$2 -async 1 -vsync -1 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 400k -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:360” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_360p.m3u8 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 500K -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:480” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_480p.m3u8 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 1500K -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:720” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_720p.m3u8 -c copy -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2.m3u8 &amp;<br>elif [ “${height}” -lt 721 ] &amp;&amp; [ “${height}” -gt 480 ]<br>then <br>ffmpeg -i rtmp://172.104.55.76/$1/$2 -async 1 -vsync -1 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 400k -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:360” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_360p.m3u8 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 500K -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:480” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_480p.m3u8 -c copy -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2.m3u8 &amp;<br>elif [ “${height}” -lt 481 ] &amp;&amp; [ “${height}” -gt 300 ]<br>then<br>ffmpeg -i rtmp://172.104.55.76/$1/$2 -async 1 -vsync -1 -c:v libx264 -x264opts keyint=24:no-scenecut -c:a aac -max_muxing_queue_size 4000 -r 30 -b:v 400k -profile:v high -b:a 128k -vf “trunc(oh*a/2)*2:360” -tune zerolatency -preset veryfast -crf 23 -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2_360p.m3u8 -c copy -f hls -hls_time 1 -hls_playlist_type event -hls_list_size 10 /mnt/hls/$2.m3u8 &amp;<br>else <br>echo “not working”<br>fi wait</strong></p><p>It downgrades the resolutions and create seperate HLS files. To read about the ffmpeg flags you can visit the official site of ffmpeg there are lots of things to learn about the library. <a href="https://www.ffmpeg.org/"><strong>https://www.ffmpeg.org/</strong></a><strong>. </strong>With this setup with slight modification you can create a successfully streaming server, I hope it gives you a clear understanding to stream hls or mpeg-dash you can use hls.js or mpeg-dash.js or there are more libraries like flowflv to stream rtmp directly. Next time will show how to create own rtmp server using the adobe RFC till then enjoy and keep learning…. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f46dea0829fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Websocket Under The Hood]]></title>
            <link>https://sudeepdasgupta.medium.com/webocket-under-the-hood-cab706c7f24?source=rss-85a15b01d7ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/cab706c7f24</guid>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[scratch]]></category>
            <category><![CDATA[rfc]]></category>
            <category><![CDATA[tcp]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Sudeep Dasgupta]]></dc:creator>
            <pubDate>Sat, 30 Nov 2019 10:34:43 GMT</pubDate>
            <atom:updated>2021-05-19T18:14:11.079Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Web Socket Under The Hood</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*ZneDqeqhmGYCre0fUJFQzQ.png" /></figure><p>Image is taken from Azure Credit goes to Microsoft for the image.</p><p>Basic understanding of Web Socket. What is Web Socket and how it differs from TCP sockets So Web Socket is actually TCP sockets but it establishes handshake in a more secure and different way? It works on an upgrade of HTTP 1.1 or above Protocol. Where the browser or any Web Socket client request to the HTTP server for an upgrade. The server sends a header with a 101 switching protocol status code, HTTP status code to the client and a handshake happens on basis of the Web Socket key that is included in the header (“ <strong>sec-websocket-key</strong>”). You can find the HTTP Status Code and headers in the following link. <a href="https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol"><strong>https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol</strong></a>. This is based on the RFC <a href="https://tools.ietf.org/html/rfc6455">6455</a> and following is the link: <a href="https://tools.ietf.org/html/rfc6455"><strong>https://tools.ietf.org/html/rfc6455</strong></a></p><p>Now, this example is not meant for production, it is just an example where you will be learning about WebSocket fundamentals and how connections are established. We will be using NodeJS, the same can be implemented in other languages also with the same steps whether it&#39;s PHP, JAVA, c++, golang, or any other. To find the full code you can visit my git repo: <a href="https://github.com/ignition123/WebSocket_Underthehood"><strong>https://github.com/ignition123/WebSocket_Underthehood</strong></a></p><p>Step1: We will import four modules of NodeJS. 2 of them are not needed actually to write in a file and to show the Endianness of your CPU. To know about what is Endian(Big/Little) read the following link which is very useful to work on bytes. <a href="https://en.wikipedia.org/wiki/Endianness"><strong>https://en.wikipedia.org/wiki/Endianness</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*7zvV2daSEh6Hlf1JNcYlgg.png" /></figure><p>Here we have import HTTP module, crypto module, file system module, and os module which are in build module of NodeJS.<br>Now, let&#39;s print the endianness of the CPU.<br>To do this you can write: <strong>console.log(os.endianness());</strong> // LE / BE<br>Now to start the HTTP server, we will call the createServer method:<br><strong>let server = http.createServer();</strong><br>and listen to the server in port 9100, here you can put any port you want.<br><strong>server.listen(9100);.<br></strong>Now we will call en event <strong>upgrade</strong> event and you can put an event listener that is again upto you.<br><strong>server.addListener(“upgrade”,function(req,socket){});</strong></p><p>Now inside this method we will call some more events and some encryption. Now here we will create a header. Here we have used <strong>SHA1</strong> encryption with <strong>BASE64 </strong>encoding on one of the header <strong>sec-websocket-key </strong>and concatenated with a magic key that is given in RFC<strong> “258EAFA5-E914–47DA-95CA-C5AB0DC85B11”.<br>let respHeader = crypto.createHash(‘sha1’).update(req.headers[“sec-websocket-key”]+”258EAFA5-E914–47DA-95CA-C5AB0DC85B11&quot;).digest(‘base64’);<br></strong>Now we will prepare response string to send it to the client.<br><strong>let response = “HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Accept: “+respHeader+”\r\n\r\n”;<br></strong> <strong>socket.write(response); <br>socket.setNoDelay(true);<br>socket.setKeepAlive(true);<br></strong>Now we will initialize a class to handle the raw data and parse it, we will declare the class in the bellow. To Initialize the class we write. Again this is a demo for learning purposes to make it for production you can use event emitter and another process to make better code. Where we can emit the events on parsing.<br><strong>let msgParse = new SocketParser(socket);</strong><br>Now we will call three more events of TCP that are given in NodeJS documentation. To know about more events and methods you can visit to nodejs official site. <br><a href="https://nodejs.org/api/net.html"><strong>https://nodejs.org/api/net.html</strong></a><strong><br>socket.addListener(“data”,function(buffer) { <br> console.log(buffer); <br> msgParse.addBuffer(buffer); <br>}); <br>socket.addListener(“end”,function() {<br> </strong>// here you can handle on socket disconnect from either client or server side.<strong><br>}); <br>socket.addListener(“error”,function(err) {<br></strong> // here you can handle if any tcp error occurs<strong><br>});<br></strong>Here we have called a method <strong>“msgParse.addBuffer(buffer);”</strong> here we will handle the raw bytes buffers that we receive from TCP sockets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3M00nvz_Z54p8nYyMvwxw.png" /></figure><p>Now we will define the body of the SocketParser class. First, we will call</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*zpYUYiNnSquusg9GI8iU_w.png" /></figure><p>Now we will create a method called startTimer, this method will trigger when the first byte is received. This is a very poor way to handle message but again this is just for demo purpose to see how the parsing happens you can use event handlers and while loop to parse messages continuously rather than serInterval. This method will trigger every 10ms and call a method named GetPayload which we will define later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*bQoqjNjyDSjKF_V471JWbg.png" /></figure><p>Now we write body of another method that is addBuffer, which will be invoked everytime a buffer is recieved in <strong>on(“data”) </strong>event. Now what this method is doing, it is appending the buffer into an array <strong>“this.buffer”</strong> and then getting the first buffer to another variable call “<strong>this.curPacket” </strong>using shift method. If this.curPacket is empty then directly putting the first buffer into it or else it will concatenating the old the one. To know about buffer you can go to nodejs official website for buffer package.<br><strong>addBuffer(buffer) { <br></strong>// if the buffer array is empty it will start the timer<strong> <br>if(this.buffer.length === 0) { this.startTimer(); } <br></strong>// it pushes all the buffers into an array that are coming from on data event this.buffer.push(buffer); <strong><br></strong>// if current packet length is 0 then the current packet as only the current buffer <br>// if the current packet length is not 0 then it append the latest buffer with the old one <strong><br>if(this.curPacket.length === 0) { <br>this.curPacket = Buffer.concat([this.buffer.shift()]); <br>} else { <br>this.curPacket = Buffer.concat([this.curPacket, this.buffer.shift()]); } <br>}</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*e8l4RE1MYUwMJQW7tB5pUQ.png" /></figure><p>Now we will define the body of the GetPayload method. It is very simple and handles some cased of WebSocket not all cases but enough for you to receive and send messages of both text and binary data.<br>Read the line by line comments to understand the meaning of the code.<br><strong>GetPayload(){</strong><br><em>// Here it is the checking if the this.curPacket is empty, then it will clear the interval<br></em><strong>if(<em>this</em>.curPacket.length === 0){<br>clearInterval(<em>this</em>.timer);<br>return;<br>}</strong><br><em>// Here it is checking the the this.prevPacketText is empty string or not, this method is invoked when there is fin 1 without the occurance of previous message with fin 0<br></em><strong>if(<em>this</em>.prevPacketText === “”){</strong><br><em>// now we will get the first bit, to know bit and bytes read the following link.<br>// </em><a href="https://www.javatpoint.com/java-data-types"><em>https://www.javatpoint.com/java-data-types</em></a><em><br>// 8 bits = 1 byte<br>// first byte of the buffer<br></em><strong><em>var</em> firstByte = <em>this</em>.curPacket[0];</strong><br><em>// getting the first bit the fetch the fin 0x80 is hex representation of decimal you can read hex conversion from google. &amp; and &gt;&gt;&gt; these are bit operators you can check this in javascript documentation. These are basics of javascript.<br></em><strong><em>this</em>.prevfin = (firstByte &amp; 0x80) &gt;&gt;&gt; 7;</strong><br><em>// getting the opCode from the second bit of the first byte<br></em><strong><em>this</em>.prevOpCode = firstByte &amp; 0x0f;</strong><br><em>// here we are handling special opCode that we want to ignore for now.<br></em><strong><em>var</em> payloadType;<br>if ( <em>this</em>.prevOpCode &gt;= 0x3 &amp;&amp; <em>this</em>.prevOpCode &lt;= 0x7<br>||<br><em>this</em>.prevOpCode &gt;= 0xB &amp;&amp; <em>this</em>.prevOpCode &lt;= 0xF)<br>{<br>console.log(“Special frame recieved”);<br>return;<br>}</strong><br><em>// now matching the opCode and setting payloadType string<br>// if we recieve 0 that means packets are coming more and we need to continue to read the packets before parsing.<br>// if we recieve 1 that means text and 2 means binary data<br>// if we recieve 9 means client / server has disconnected and sends the opCode to other party<br>// if we recieve ping then server will send pong to maintain heartbeat same goes for client.<br></em><strong>if(<em>this</em>.prevOpCode == 0x0){<br>payloadType = ‘continuation’;<br>}else if(<em>this</em>.prevOpCode == 0x1){<br>payloadType = ‘text’;<br>}else if(<em>this</em>.prevOpCode == 0x2){<br>payloadType = ‘binary’;<br>}else if(<em>this</em>.prevOpCode == 0x8){<br>payloadType = ‘connection close’;<br>}else if(<em>this</em>.prevOpCode == 0x9){<br>payloadType = ‘ping’;<br>}else if(<em>this</em>.prevOpCode == 0xA){<br>payloadType = ‘pong’;<br>}else{<br>payloadType = ‘reservedfornon-control’;<br>}</strong><br><em>// here we are printing the fin and opCode<br></em><strong>console.log(“this.prevfin: “+<em>this</em>.prevfin);<br>console.log(payloadType);</strong><br><em>// Here we are handling continuation flag and we are parsing the values and more packets are about to come.<br></em><strong>if(payloadType === “continuation”){<br><em>this</em>.parseMessage(<em>this</em>.prevfin);<br>return;<br>}</strong><br><em>// here if the fin = 1 and payloadType is text or binary means message is complete<br>// if fin is 0 then we store the packet after parse and wait for next packet to parse and then we concatenate the packets to one.<br></em><strong>if(<em>this</em>.prevfin === 1 &amp;&amp; (payloadType === “text” || payloadType === “binary”)){<br><em>this</em>.parseMessage(<em>this</em>.prevfin);<br>}else if(<em>this</em>.prevfin === 0){<br><em>this</em>.parseMessage(<em>this</em>.prevfin);<br>}</strong></p><p><strong>}else{</strong><br><em>// this case comes when previous fin = 0 and we store the packet to </em><strong><em>this.prevPacketText and then we parse the current packet and concatenate both.<br></em>if(<em>this</em>.prevfin === 1){<br><em>this</em>.parseMessage(1, &quot;append&quot;);<br>}else{<br><em>this</em>.parseMessage(<em>this</em>.prevfin);<br>}</strong><br><strong>}<br>}</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FqIUxKn-EF3sxEetZYS3rw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d6Ll9DqY_LluSTiKhWMSSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4YHd4qiypNW2G53cl_jZOA.png" /></figure><p>Here we have played with the first byte of the packet. In websocket we get fin, opCode, maskBool which says if packet is masked or not, if the packet is not masked the server should not parse the packets and disconnects the clients. The client should always send messages in XOR encryption. To read about XOR encryption see the following link. <a href="https://www.geeksforgeeks.org/xor-cipher/">https://www.geeksforgeeks.org/xor-cipher/</a>. Then we recieve the payloadLength, mask flag which is of 4 bytes size. Now we will write a method to parse the message this is the main method where we parse the raw buffers.<br><strong>parseMessage(<em>finstatus</em>, <em>state</em>){</strong><br><em>// Here we get the second bytes<br></em><strong><em>var</em> secondByte = <em>this</em>.curPacket[1];</strong><br><em>// first bit of the second bytes where we get the masking flag whether the message is masked or not, if not disconect the client. Masking means the data should be in XOR encryption<br></em><strong><em>var</em> mask = (secondByte &amp; 0x80) &gt;&gt;&gt; 7;</strong><br><em>// if mask === 0 then get the next packet or disconect the client.<br></em><strong>if (mask === 0){<br><em>this</em>.status = true;<br>console.log(‘browse should always mask the payload data2’);<br>return;<br>}</strong><br><em>// get the payload length of the buffer if it is 125 then boom thats it you dont have to get the packet length, and directly unmask the data,<br>// if the payload Length is 126 then you need to read 2 bytes to get the actual payload length that is short value of 2 bytes.<br>// if the payload length is 127 then message is very huge and is of 64 bit message that is 8 bytes to get the actual packet size.<br></em><strong><em>var</em> payloadLength = (secondByte &amp; 0x7f);<br><em>var</em> decoded = [];<br><em>var</em> text = “”;<br><em>var</em> offset = 2;</strong><br><em>// if packet length is 126 then slice 2 bytes from the current packet and read convert the buffer to number to get the actual payload size<br>// if the packet lenth is 127 then slice 8 bytes from the current packet and read convert the buffer to number to get the actual payload size<br></em><strong>if(payloadLength == 126){<br>payloadLength = <em>this</em>.curPacket.slice(offset, 4);<br>payloadLength = payloadLength.readUInt16BE(0);<br>offset += 2;<br>}else if(payloadLength == 127){<br>payloadLength = <em>this</em>.curPacket.slice(offset, 16);<br>payloadLength = payloadLength.readUInt64BE(0);<br>offset += 8;<br>}</strong><br><em>// here we are printing the payload lenth and the current packet length<br></em><strong>console.log(“payloadLength:”,payloadLength);<br>console.log(“currentBuffer:”,<em>this</em>.curPacket.length);</strong><br><em>// here we are reading more 4 bytes to get the mask buffer<br></em><strong><em>var</em> masks = <em>this</em>.curPacket.slice(offset, (offset + 4));<br><em>var</em> actualPayload = offset + 4;</strong><br><em>// getting the actual payload starting point<br></em><strong><em>var</em> totalPayloadChunk = actualPayload + payloadLength;</strong><br><em>// if the current Packet size is less than payload lenth then we return it from here and wait for more packets to recieve<br></em><strong>if(payloadLength &gt; <em>this</em>.curPacket.length){<br>return;<br>}</strong><br><em>// now we iterate the actual payload to total payload chunk size and decode the payload with the mask buffer<br></em><strong>for (<em>var</em> i = actualPayload, j = 0; i &lt; totalPayloadChunk; i++,j++){<br>decoded[j] = <em>this</em>.curPacket[i] ^ masks[j % 4];<br>text += String.fromCharCode(<em>this</em>.curPacket[i] ^ masks[j % 4]);<br>}</strong><br><em>// after the parsing the buffer now we slice the read packet from the actual buffer packet using slice<br></em><strong><em>this</em>.curPacket = <em>this</em>.curPacket.slice(totalPayloadChunk);</strong><br><em>// if state is append that means we got fin 0 in previous packet and have stored the parsed packet into this.prevPacketText and second packet we have recieved the fin 1 and concatenate the previous with the current packet<br></em><strong>if(state === “append”){<br>text = <em>this</em>.prevPacketText + text;<br><em>this</em>.prevPacketText = “”;<br>decoded = <em>this</em>.prevPacketDecoded.concat(decoded);<br><em>this</em>.prevPacketDecoded = [];<br>}</strong><br><em>// if fin status == 0 then we are storing the parsed packet into his.prevPacketText and this.prevPacketDecoded for binary values<br></em><strong>if(finstatus === 0){<br><em>this</em>.prevLen = payloadLength;<br><em>this</em>.prevPacketText += text;<br><em>this</em>.prevPacketDecoded.concat(decoded);<br>return;<br>}</strong><br><em>// here we finally writing to a file to see whether we have recieved the exact packet from the client or not<br></em><strong>fs.appendFileSync(“sample.txt”, text.length+”\r\n”);</strong><br><em>// Now we will implement the send method here.<br>// We will send message from server to client<br></em><strong><em>var</em> message = “Hello from server”;</strong><br><em>// getting the message length<br></em><strong><em>var</em> messageSize = message.length;</strong><br><em>// creating two array one for creating headers and one for creating body<br></em><strong><em>var</em> header = [];<br><em>var</em> sendData = [];</strong><br><em>// setting the opCode for sending the message to client this variable is integer 129 according to rfc<br></em><strong><em>var</em> b1 = 0x80 | (0x1 &amp; 0x0f);</strong><br><em>// setting to first byte of the header<br></em><strong>header[0] = b1;</strong><br><em>// if message length is less that or equal to 125 then we set the message size in the first byte of the header<br>// if message lenth is greater than 125 and less than 65536 then we set the message length into the second and third byte of the header that is Short value<br>// if message length is greater than equal to 65536 that is 64 bit message size then we write the byte size into the 8 bytes of the header<br></em><strong>if(messageSize &lt;= 125){<br>header[1] = messageSize;<br>}else if(messageSize &gt; 125 &amp;&amp; messageSize &lt; 65536){<br>header[1] = 126;<br>header[2] = (messageSize &gt;&gt; 8) &amp; 255;<br>header[3] = (messageSize) &amp; 255;<br>}else if(messageSize &gt;= 65536){<br>header[1] = 127;<br>header[2] = ( messageSize &gt;&gt; 56 ) &amp; 255<br>header[3] = ( messageSize &gt;&gt; 48 ) &amp; 255<br>header[4] = ( messageSize &gt;&gt; 40 ) &amp; 255<br>header[5] = ( messageSize &gt;&gt; 32 ) &amp; 255<br>header[6] = ( messageSize &gt;&gt; 24 ) &amp; 255<br>header[7] = ( messageSize &gt;&gt; 16 ) &amp; 255<br>header[8] = ( messageSize &gt;&gt; 8 ) &amp; 255<br>header[9] = ( messageSize ) &amp; 255<br>}</strong><br><em>// now converting the message into ascii value this is not necessary.<br></em><strong>for(<em>var</em> i=0;i&lt;messageSize;i++){<br>sendData[i] = message.charCodeAt(i);<br>}</strong><br><em>// concatenating the header with the body and converting it to buffer and sending it to the client thats it Kaboom, here we have build a successfully websocket for sending and recieving the values.<br>// There is lot of things to improve, like handling the message in event based creating seperate method to send data handling multiple sockets even create custom protocol like websocket over tcp or udp.<br></em><strong><em>var</em> arr = header.concat(sendData);<br><em>var</em> body = Buffer.from(arr);<br><em>this</em>.socket.write(body);<br>}</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLu_nfi2Pps8FOeC5fqs0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yGDVbAa19puCoHHi2Rp0IQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LAEafoM5KG77lkL7jDKQSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XLwVFJHoeOWTVIRR4MdWLg.png" /></figure><p>This is now completed a fully functional WebSocket sample that is able to read and write messages, I hope you liked it, next time will be demonstrating how to create custom protocol using TCP/UDP and how that gonna help in creating pub/sub or databases.</p><p>Till then happy learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cab706c7f24" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>