<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Christos Petropoulos on Medium]]></title>
        <description><![CDATA[Stories by Christos Petropoulos on Medium]]></description>
        <link>https://medium.com/@slysterous?source=rss-c63935517ba0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*m-MzzIwNVcIn6l66HwPFSQ.jpeg</url>
            <title>Stories by Christos Petropoulos on Medium</title>
            <link>https://medium.com/@slysterous?source=rss-c63935517ba0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:05:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@slysterous/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Numeral Systems in Go]]></title>
            <link>https://slysterous.medium.com/numeral-systems-in-go-5ac9ffd0c360?source=rss-c63935517ba0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ac9ffd0c360</guid>
            <category><![CDATA[numerals]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[numeral-system]]></category>
            <dc:creator><![CDATA[Christos Petropoulos]]></dc:creator>
            <pubDate>Mon, 19 Apr 2021 17:35:41 GMT</pubDate>
            <atom:updated>2021-04-20T18:12:31.259Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pRndJHXaY43N7a7LUZ4G_w.jpeg" /><figcaption>photo by <a href="https://www.pexels.com/@logankir">https://www.pexels.com/@logankir</a></figcaption></figure><p>A <strong>numeral</strong> is a symbol or group of symbols that represents a number. <strong>Numerals are not the same as numbers just as words are not the same with the things they refer to.</strong> The symbols “11(decimal)”, “1011(binary)” and “B(hexadecimal)” are <strong>different</strong> numerals, all representing the <strong>same</strong> number. The number is the idea, the numeral is anything that represents that idea.</p><p>A positional numeral system denotes usually the extension to any <a href="https://en.wikipedia.org/wiki/Radix">base</a> of the <a href="https://en.wikipedia.org/wiki/Hindu%E2%80%93Arabic_numeral_system">Hindu–Arabic numeral system</a> (or <a href="https://en.wikipedia.org/wiki/Decimal">decimal system</a>). In a more general sense, a positional system is a numeral system in which the <strong>contribution of a digit</strong> to the value of a number is the <strong>value of the digit multiplied by a factor determined by the <em>position</em> of the digit</strong>.</p><p>In modern positional systems, such as the <a href="https://en.wikipedia.org/wiki/Decimal">decimal system</a>, the <strong><em>position</em></strong> of the digit means that its value must be multiplied by some value: in 555, the three identical symbols represent five <strong>hundreds</strong>, five <strong>tens</strong>, and five <strong>units</strong>, respectively, due to their different <em>positions</em> in the digit string.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*bXQb00XiL0am9VbiptYRDw.png" /><figcaption>Positional numeral systems glossary</figcaption></figure><p>According to its <strong>position</strong> of occurrence in the number, each digit is <strong>weighted</strong>. Towards the left, the weights increase by a constant factor equivalent to the <strong>base</strong> or <strong>radix</strong>. With the help of the radix point (‘.’), the positions corresponding to integral weights (1) are differentiated from the positions corresponding to fractional weights (&lt;1).</p><p>Any integer value that is greater than or equal to two can be used as the base or <strong>radix</strong>.</p><p>The digit position ’<strong>n</strong>’ has weight <strong>r ^n</strong>. <strong>The largest value of digit position is always 1 less than the base value</strong>. <strong>The value of a number is a weighted sum of its digits.</strong></p><p>For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*qH19bVkU6L-WHcpXzslKMQ.png" /><figcaption>Decimal (0–9 digits with radix 10) 2056 value breakdown</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*UrBcqfbQth_slc3_VOATgg.png" /><figcaption>Hexadecimal(0–9 &amp; A-F digits with radix 16) 171B value breakdown</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*JrIpqe-RX5KgD-P6nQSyQw.png" /><figcaption>The general form of value breakdown, where b is the base or radix of the numeral system</figcaption></figure><h3>The Idea</h3><blockquote><a href="https://github.com/slysterous/numeral">Quick Link</a> to package</blockquote><p>I wanted to be able to <strong>implement any numeral system</strong> and play around with <strong>the representation of numbers</strong> without having to worry about the semantics of each system. At the end of the day, as we mentioned before <strong>multiple different numerals</strong> can represent the <strong>same number</strong>. So even operations between numerals from different systems shouldn’t be an issue when they can be translated to represent the same numbers.</p><p>For some reason, an image from the movie <a href="https://www.imdb.com/title/tt2084970/">Imitation Game</a> flashed in my head. The <strong>beloved moment in human history </strong>where <a href="https://en.wikipedia.org/wiki/Alan_Turing">Alan Turing’s</a> machine, the <a href="https://en.wikipedia.org/wiki/Bombe">bombe</a>, came to life and the mechanical rotors started spinning.</p><figure><img alt="The Bombe Machine" src="https://cdn-images-1.medium.com/max/946/1*9BtkzjA9rbUgV_1_8ptt8g.jpeg" /><figcaption>A wartime picture of a <a href="https://en.wikipedia.org/wiki/Bletchley_Park">Bletchley Park</a> Bombe</figcaption></figure><p>A digit in any positional numeral system can be <strong>represented</strong> by the<strong> possible states </strong>it can take. Each state has a <strong>next state</strong> and a <strong>previous state</strong>. The <strong>next state</strong> of the <strong>last state</strong> is the <strong>first state </strong>while producing a <strong>carry-over</strong>.</p><p>A data structure like a <a href="https://www.geeksforgeeks.org/doubly-circular-linked-list-set-1-introduction-and-insertion/">circular list</a> can effectively represent what we mentioned in the above paragraph.</p><p><strong>In a</strong> <a href="https://www.geeksforgeeks.org/doubly-circular-linked-list-set-1-introduction-and-insertion/"><strong>doubly-linked circular list</strong></a>, <strong>the next pointer of the last node points to the first node, and the previous pointer of the first node points to the last node making the circular in both directions</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*xll8uTdTaK9vflthl-N4aQ.png" /><figcaption>doubly-linked circular list</figcaption></figure><p>Here comes Go’s <a href="https://golang.org/pkg/container/ring/"><strong>ring</strong></a>. With the ring, we can easily represent <strong>any single digit</strong> of <strong>any positional numeral system</strong> by filling it with the possible ordered states and an initial value.</p><pre>// newDigit creates and initializes a new digit (ring).<br>func newDigit(values []rune, state rune) (*ring.Ring, error) {<br>   // initialize a new empty ring<br>   r := ring.New(len(values))<br><br>   // fill the ring with values<br>   for _, e := range values {<br>      r.Value = e<br>      r = r.Next()<br>   }<br><br>   if indexOf(state, values) == -1 {<br>      return nil, fmt.Errorf(&quot;invalid digit. value: %v does not exist in possible values: %v&quot;, state, values)<br>   }<br><br>   // roll the ring in desired &quot;state&quot; position.<br>   for range values {<br>      if r.Value == state {<br>         break<br>      }<br>      r = r.Next()<br>   }<br>   return r, nil<br>}</pre><p>What happens when a numeral contains more than one digit?<br>That can easily be solved by having a <a href="https://www.geeksforgeeks.org/doubly-linked-list/"><strong>doubly-linked list</strong></a><strong> </strong>whose elements would be consisted of <a href="https://www.geeksforgeeks.org/doubly-circular-linked-list-set-1-introduction-and-insertion/"><strong>doubly-linked circular list</strong></a><strong>s.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*M8OI5w8cS4idbtYyr-iNag.png" /><figcaption>doubly-linked list</figcaption></figure><p>This way we can easily create <strong>any</strong> positional representation of any number.</p><pre>// NewNumeral initializes a numeral by providing the initial number in strings<br>// along with the possible values that each digit can have.<br>func NewNumeral(values []rune, initial string) (*Numeral, error) {<br>   // initialise a new number.<br>   number := Numeral{<br>      digits:      list.New(),<br>      digitValues: values,<br>   }<br>   // add digits to the number along with their state.<br>   for i := 0; i &lt; len(initial); i++ {<br>      digit, err := newDigit(values, rune(initial[i]))<br>      if err != nil {<br>         return nil, err<br>      }<br>      number.digits.PushBack(digit)<br>   }<br>   return &amp;number, nil<br>}</pre><p>Every time a <strong>digit</strong> resets back to the “first” state it would trigger a <strong>Next()</strong> to the next most valuable digit. Pretty simple right? <strong>In the case that our digits are all full and we want to increment, we can just create a new ring, initialize it and link it to the left side of our list.</strong></p><p>Think of an example, a hypothetical numeral system with <strong>symbols [0–9,a-z]</strong> which has a <strong>radix of 36</strong>, and also imagine that we want to have a 4 “digit numeral” like “2z3r”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1gccYqFquooG892gBw8hWw.png" /><figcaption>visual representation of <strong>doubly-linked list</strong> of <strong>doubly-linked circular list</strong></figcaption></figure><pre>// Numeral represents a numeral that is consisted by its digits<br>// and digit values.<br>type Numeral struct {<br>   digits      *list.List<br>   digitValues []rune<br>}</pre><p>By defining a number like this all you need to have is a <strong>slice of the symbols</strong> you want to <strong>represent</strong> your number <strong>in the order they will have to exist</strong>.</p><pre>var values= []rune{&#39;0&#39;, &#39;1&#39;, &#39;2&#39;, &#39;3&#39;, &#39;4&#39;, &#39;5&#39;, &#39;6&#39;, &#39;7&#39;, &#39;8&#39;, &#39;9&#39;,<br>   &#39;a&#39;, &#39;b&#39;, &#39;c&#39;, &#39;d&#39;, &#39;e&#39;, &#39;f&#39;, &#39;g&#39;, &#39;h&#39;, &#39;i&#39;, &#39;j&#39;, &#39;k&#39;, &#39;l&#39;, &#39;m&#39;, &#39;n&#39;, &#39;o&#39;, &#39;p&#39;, &#39;q&#39;, &#39;r&#39;, &#39;s&#39;, &#39;t&#39;, &#39;u&#39;, &#39;v&#39;, &#39;w&#39;, &#39;x&#39;, &#39;y&#39;, &#39;z&#39;}</pre><p>Then you need a “constructor” to generate numerals at will.</p><pre>// NewNumeral initializes a numeral by providing the initial number in strings<br>// along with the possible values that each digit can have.<br>func NewNumeral(values []rune, initial string) (*Numeral, error) {<br>   // initialise a new number.<br>   number := Numeral{<br>      digits:      list.New(),<br>      digitValues: values,<br>   }<br>   // add digits to the number along with their state.<br>   for i := 0; i &lt; len(initial); i++ {<br>      digit, err := newDigit(values, rune(initial[i]))<br>      if err != nil {<br>         return nil, err<br>      }<br>      number.digits.PushBack(digit)<br>   }<br>   return &amp;number, nil<br>}</pre><p>Initializing a numeral after we have reached this point is pretty straightforward.</p><pre>number, _ := numeral.NewNumeral(values, &quot;2z3r&quot;)</pre><h3>Performing Simple Operations</h3><p>This approach makes +1 and -1 operations <strong>very fast</strong> since <strong>lists (either circular or not) </strong>always know their next and previous node.</p><p>For example:</p><ul><li>Incrementing a <strong>Hexadecimal</strong> numeral <strong>“12d4” </strong>by <strong>one</strong> needs to execute <strong>1 next() to the rightmost ring, that&#39;s all. </strong>The result is “12d5”.</li><li>Incrementing a <strong>Hexadecimal</strong> numeral “<strong>dfff</strong>” by <strong>one</strong> needs to execute <strong>4 next() calls on the 3 rightmost rings and one last on the 4th most significant ring</strong>. The result is “e000”.</li><li>Decrementing1will operate in a similar way.</li></ul><pre>// Increment performs a +1 to the Numeral.<br>func (n *Numeral) Increment() error {<br>   // take the first digit from the right and keep going if there are any arithmetic holdings.<br>   for e := n.digits.Back(); e != nil; e = e.Prev() {<br>      // get current ring.<br>      r := e.Value.(*ring.Ring)<br><br>      // rotate and update.<br>      r = r.Next()<br>      e.Value = r<br><br>      // if the digit is not being reset (no arithmetic holdings) then there is no need to<br>      // proceed in adding on the others.<br>      if r.Value != n.digitValues[0] {<br>         break<br>      }<br><br>      // If needed add an extra new digit on the left side.<br>      if e.Prev() == nil {<br>         d, _ := newDigit(n.digitValues, n.digitValues[0])<br>         n.digits.PushFront(d)<br>      }<br>   }<br>   return nil<br>}</pre><p>Adding and Subtracting can also happen efficiently even between numerals that are based on different systems by using the most known numbers of the decimal system that our computers easily understand.</p><ul><li><strong>Adding 2 numerals</strong> will just require <strong>converting</strong> both of those to <strong>decimal integers</strong>, add them and generate a new numeral from the <strong>result</strong>. The result numeral can even be on a different numeral system easily by generating it from an integer.</li></ul><p>Converting any numeral to integer is easy by performing a value breakdown</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*JrIpqe-RX5KgD-P6nQSyQw.png" /><figcaption>the general form of value breakdown, where b is the base or radix of the numeral system</figcaption></figure><pre>// Decimal converts a numeral to a decimal integer.<br>func (n *Numeral) Decimal() int {<br>   dec := 0<br>   di := 0<br>   for d := n.digits.Back(); d != nil; d = d.Prev() {<br>      // get current ring.<br>      r := d.Value.(*ring.Ring)<br>      // get the index of the ring.<br>      i := indexOf(r.Value.(rune), n.digitValues)<br><br>      // Add digit&#39;s decimal counterpart to the decimal sum.<br>      dec = dec + i*powInt(len(n.digitValues), di)<br>      di++<br>   }<br>   return dec<br>}</pre><h3><strong>Let’s Talk Numbers</strong></h3><p>Running some <strong>benchmarks</strong> on the <strong>Increment()</strong> method for <strong>3 different numeral</strong> <strong>systems</strong> for different initial values (small and big) results in some very nice results. The average execution time of the Increment function for binary numerals after 1 billion increments is around 7.6ns / op.</p><p>As one may expect, <strong>selecting a numeral system with a higher numeral range results in carryovers appearing less frequently,</strong> therefore the transitions that need to happen are <strong>fewer</strong> (increments/decrements)which results in better performance.</p><pre><br>goos: linux<br>goarch: amd64<br>pkg: github.com/slysterous/numeral<br>cpu: Intel(R) Core(TM) i7–7700 CPU @ 3.60GHz<br>BenchmarkNumeralIncrementDecimalSmall-8 1000000000 4.604 ns/op<br>BenchmarkNumeralIncrementDecimalLarge-8 1000000000 4.574 ns/op<br>BenchmarkNumeralIncrementBinarySmall-8 1000000000 7.644 ns/op<br>BenchmarkNumeralIncrementBinaryLarge-8 1000000000 7.655 ns/op<br>BenchmarkNumeralIncrementHexSmall-8 1000000000 4.427 ns/op<br>BenchmarkNumeralIncrementHexLarge-8 1000000000 4.652 ns/op</pre><p>How about converting numerals into decimal integers? From the benchmarks that I performed on Hex and Binary numerals, the process takes <strong>less than 1 μsecond</strong>. The opposite process, converting from decimal ints to any numeral is similar.</p><p>Since converting to decimals <strong>requires traversing through every digit to sum its weight</strong>, it’s easy to reach the conclusion that <strong>the more digits there are</strong>, <strong>the slower the process will be</strong>. This can also be reflected in the below benchmark.</p><pre>go test -bench=NumeralDecimal -benchtime=60s<br>goos: linux<br>goarch: amd64<br>pkg: github.com/slysterous/numeral<br>cpu: Intel(R) Core(TM) i7–7700 CPU @ 3.60GHz<br>BenchmarkNumeralDecimalFromHex0–8 1000000000 8.182 ns/op<br>BenchmarkNumeralDecimalFromHexfffff-8 252367634 284.8 ns/op<br>BenchmarkNumeralDecimalFromHexffffffffff-8 250184980 286.0 ns/op<br>BenchmarkNumeralDecimalFromHexffffffffffffffffffff-8 100000000 616.6 ns/op<br>BenchmarkNumeralDecimalFromBinary0–8 1000000000 7.818 ns/op<br>BenchmarkNumeralDecimalFromBinary10000–8 837013039 84.98 ns/op<br>BenchmarkNumeralDecimalFromBinary1000000000–8 337445121 214.8 ns/op<br>BenchmarkNumeralDecimalFromBinary1000000000000000000–8 158028769 455.9 ns/op</pre><p>Alright, one last check. How does adding numerals of different systems perform? Remember, I’m using decimal integers as the intermediate common base to perform operations. Turns out adding is also relatively fast since it <strong>takes roughly around 8 μseconds</strong> to perform an average addition</p><p>Additions are a bit trickier in terms of performance. At least 2 conventions have to happen, both numbers are translated to decimal and in the end, the result is converted back to the final desired numeral system. This <strong>relies heavily</strong> on the <strong>performance of the conversion of a numeral to decimal int</strong>.</p><pre>go test -bench=NumeralAdd -benchtime=60s<br>goos: linux<br>goarch: amd64<br>pkg: github.com/slysterous/numeral<br>cpu: Intel(R) Core(TM) i7–7700 CPU @ 3.60GHz<br>BenchmarkNumeralAddBinaryOnHex-8 9691966 8651 ns/op<br>BenchmarkNumeralAddHexOnHex-8 9144648 8290 ns/op<br>BenchmarkNumeralAddDecOnDec-8 11384712 7014 ns/op<br>BenchmarkNumeralAddSumBinaryOnHex-8 11281405 6237 ns/op<br>BenchmarkNumeralAddSumHexOnHex-8 12745245 5873 ns/op<br>BenchmarkNumeralAddSumDecOnDec-8 12672326 5849 ns/op</pre><p>In conclusion, <a href="https://github.com/slysterous/numeral">numeral</a> is now a Go package. It gives you the ability to create custom (positional) numeral systems and perform operations on them. Feel free to contribute and don’t hesitate to reach out for any questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ac9ffd0c360" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A detailed guide to the world of MQTT]]></title>
            <link>https://slysterous.medium.com/a-detailed-guide-to-the-world-of-mqtt-8155f7475dd5?source=rss-c63935517ba0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8155f7475dd5</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[mqtt]]></category>
            <category><![CDATA[broker]]></category>
            <category><![CDATA[pub-sub]]></category>
            <category><![CDATA[m2m]]></category>
            <dc:creator><![CDATA[Christos Petropoulos]]></dc:creator>
            <pubDate>Sun, 07 Jul 2019 11:52:29 GMT</pubDate>
            <atom:updated>2019-07-22T07:14:39.755Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6U3LvYLKsyE6D0Acefgj4g.png" /></figure><p>MQTT (formerly known as MQ Telemetry Transport) was <a href="https://www.ibm.com/podcasts/software/websphere/connectivity/piper_diaz_nipper_mq_tt_11182011.pdf">created</a> in 1991 by Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech) in order to connect oil pipelines over unreliable, satellite networks. It is a machine-to-machine (<a href="https://en.wikipedia.org/wiki/Machine_to_machine">M2M</a>)/”Internet of Things” connectivity protocol.</p><p><strong>It was designed as an extremely lightweight publish/subscribe messaging transport</strong>. It is mostly useful for establishing connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. The protocol is data-centric, unlike for example HTTP, which is document centric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*OkfdAUGSCiGJtPMemhpO6w.png" /></figure><p><strong>It is based on the </strong><a href="https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern"><strong>Pub/Sub</strong></a><strong> messaging pattern </strong>where clients instead of focusing on who the receiver of the message is, they focus on categorizing the messages in topics of interest. Messages are sent to an intermediate, a message broker where interested clients subscribe to.</p><p>The message structure is compact and the overhead of the binary header is just 2 bytes.</p><p><strong>Every client has to be connected to the Message Broker to interact with the system</strong>, which sometimes makes the usage of the protocol not very efficient in cases of <strong>one message duration connections</strong>. Connecting and disconnecting to the broker has a resource cost which, in certain cases, can make the usage of the protocol as efficient as other protocol alternatives. Connection utilization is important in MQTT.</p><p>At the moment the most widely used version of the protocol is MQTT v3.1.1 which is implemented by many <a href="https://github.com/mqtt/mqtt.github.io/wiki/brokers">brokers</a>. <a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html">Version 3.1.1</a> came out on November 7, 2014.</p><p>On April 3, 2019, <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">MQTT v5.0</a> became an official <a href="https://www.oasis-open.org/">OASIS</a> standard bringing a lot of new features. Some of them are:</p><ul><li>Shared Subscriptions</li><li>Session Expiration</li><li>Message Expiration</li></ul><p>For the details of this version, you can refer to this <a href="http://mqtt.org/2019/04/mqtt-v5-0-now-an-official-oasis-standard">link</a>.</p><h3>MQTT and Message Queues</h3><p><strong>The protocol is not directly related to </strong><a href="https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_8.0.0/com.ibm.mq.pro.doc/q002620_.htm"><strong>message queues</strong></a><strong>.</strong> Although some versions of the protocol can implement components that have similar behavior to queues, this is not one of the protocols described capabilities. This is usually a confusing subject. Many sources label MQTT as Message Queueing Telemetry Transport which is not correct. “MQ” refers to the MQ Series, an IBM product developed to support MQ telemetry transport. When the protocol was created, it was named after the IBM product.</p><p>In message queues, messages are <strong>stored in a queue</strong> until a consumer gets the message. In MQTT, messages that arrive in a topic without any subscribers are not kept in queues (unless a persistent session is enabled where all QOS1 &amp; QOS2 messages, that are not yet confirmed or missed, are stored until consumption).</p><p>Messages in queues can be consumed by one client. In MQTT a message can have multiple subscribers. <strong>Message Queuing and Pub/Sub Messaging are two completely different messaging patterns.</strong></p><p>The biggest difference with queues is that they need to be explicitly created and they also must be named. This is not the case with MQTT, where topics are flexible and created on the fly under a tree-like structure.</p><p>MQTT can provide a type of message queues under certain circumstances. This can happen when a persistent session is present between a client and the broker.</p><p>If a client that has a persistent session is disconnected from the broker and at some point reconnects, the following information is available immediately:</p><ul><li>All the subscriptions of a client</li><li>All QOS1 &amp; QOS2 flow messages that have not been confirmed</li><li>All new QOS1 &amp; QOS2 messages that the client missed</li><li>All QOS2 messages received from the client and are not yet completely acknowledged</li></ul><p>More information regarding persistent sessions can be found <a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349231">here</a>.</p><h3>Protocol Characteristics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*r5eWIiFYIoGT6k0uOjLMUg.png" /></figure><h4>Simplicity</h4><p>By design, the MQTT protocol focuses on simplicity and mainly exposes 5 ways of interaction.</p><ul><li><strong>Connect</strong> — <em>Client waits for a connection to be established with the message broker.</em></li><li><strong>Disconnect</strong> — <em>Broker waits for the MQTT client to finish any work, which needs to be done for the </em><a href="http://searchnetworking.techtarget.com/definition/TCP-IP"><em>TCP/IP</em></a><em> session to disconnect.</em></li><li><strong>Subscribe</strong> — <em>Client Requests to subscribe to one or more topics.</em></li><li><strong>Unsubscribe</strong> — Client <em>Requests to unsubscribe from one or more topics.</em></li><li><strong>Publish</strong> <strong>— </strong><em>Client or Broker, publish a message to a topic.</em></li></ul><h4>Multiple Service Levels (QoS Quality of Service)</h4><p>Message delivery can be achieved in <a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349835"><strong>three</strong></a> different levels by the protocol which can be very convenient depending on the use case. QoS can be different for subscribing and publishing. Below we can see how the different levels work. <strong>NOTE</strong>: Entities can either refer to any client or the broker.</p><p><strong>QOS0</strong></p><p>The message will be published or received <strong>at most once</strong> (fire and forget).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*wKSwpdoEFs5vBpIjVJ7NXg.png" /></figure><p><strong>QOS1</strong></p><p>The message will be published or received <strong>at least once, </strong>meaning that the sender stored the message and will probably repeat the procedure if he does not receive an acknowledgment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*N3MCrHsROltPcUf4RMm_qA.png" /></figure><p><strong>QOS2</strong></p><p>The message will be published or received <strong>exactly one time </strong>and will include an extra verification step for the communicated message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*BmiNMdiOQMkO5jEcrVW-2A.png" /></figure><p><strong>What is really worth making crystal clear, is that the involved entities in each communication scenario are three,</strong> the publisher, the broker, and the subscriber.</p><p><strong>Publisher and subscriber are not designed to use the same type of service</strong>(QoS). Each client communicates only with the intermediate, the broker. <strong>Clients are not destined to know the existence of one another</strong>.</p><p>Let’s analyze an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*XGBbafIt2_wFyharMvtSjg.png" /></figure><p>Say we have a <strong>smart thermometer</strong> that connects to the message broker and <strong>publishes its temperature</strong> in a topic every 10 seconds in a “<strong>fire and forget</strong>” type of service. An <strong>app</strong> for example on the other side <strong>has subscribed to the temperature topic, </strong>in an “<strong>at least once</strong>” type of service, and displays the received temperature on the screen.</p><p>In this example, the smart thermometer has a very small network bandwidth footprint since it is publishing in a <strong>fire and forget</strong> manner. In most cases, temperatures do not tend to drastically change in the time frame of 10 seconds so QOS0 seems to be the most appropriate type to use.</p><p>The app, on the other hand, should display <strong>any</strong> differences in temperature, so it should receive any messages <strong>at least once</strong>.</p><p>This summarizes our design and brings us to the conclusion that involved entities, always depending on the case, should not use the same type of service.</p><h4>One To ∞ Message Distribution &amp; Topic Structure</h4><p><strong>Each client that publishes a message on the message broker, by design, has no awareness that anyone other than the broker has received the message</strong>. Messages could be identified as interests or topics of discussion that anyone interested, could receive knowledge of.</p><p>What I find particularly funny is that when a client subscribes to a topic, it has no knowledge, whether or not, the particular topic actually exists or has (LWT) or will have any data in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*qavmufBf7IVFxk9HAeChwA.png" /></figure><p>This reminds me of the <a href="https://simple.wikipedia.org/wiki/Schr%C3%B6dinger%27s_cat"><strong>Schrödinger’s cat thought experiment</strong></a>, but instead of the cat, we are referring to the data that exist or will exist in a topic.</p><p>The only way to find out about them is to subscribe to a topic.</p><p>Topics that exist or will exist can have multiple or no subscribers at all.</p><p><strong>MQTT topic structure is often mentioned as the topic tree</strong>. It is a tree-like structure that is split into levels. Topics defined by clients are UTF-8 strings that are processed by the broker to filter messages. Each topic consists of one or more topic levels. Topic levels are separated by forward-slashes “/”.</p><p>When subscribing to a topic, there are specific characters that can be used as wildcards. MQTT supports two types of wildcards. The single-level wildcard “<strong>+</strong>” and the multi-level wildcard “<strong>#</strong>”.</p><p>Let’s take the following topic tree as an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*wzm4ihR2JSfGkrN_6tZUJA.jpeg" /></figure><p>If we wanted to subscribe to messages regarding the humidity of the kitchen, all we would have to do is subscribe to:</p><pre>home/kitchen/generic/humidity</pre><p>What if we wanted to subscribe to the temperature of all the available rooms? We would use the single level wildcard character.</p><pre>home/+/generic/temperature</pre><p>Last but not least, where would we subscribe to, if we wanted to listen to all messages regarding the kitchen and everything that is in there? This time we would use the multi-level wildcard character.</p><pre>home/kitchen/#</pre><p>There are some best practices regarding topic structure that we will discuss later on.</p><h4>Compact Packet</h4><p>One of MQTT’s main strengths is the small overhead it adds to communications. Its packet headers are compact and have a binary structure.<br>They consist of 3 parts at most.</p><ul><li><strong>Fixed header</strong>, present in all MQTT Control Packets</li><li><strong>Variable header</strong>, present in some MQTT Control Packets</li><li><strong>Payload</strong>, present in some MQTT Control Packets</li></ul><p>The <strong>Fixed header</strong> is always present in each MQTT packet. It is two bytes in size. The detailed structure of the fixed header is given below.</p><p>The <strong>Variable header</strong> is not present in all MQTT control packets. Its structure is different for different MQTT requests.</p><p><strong>Payload</strong> is the actual data which is going to be sent. It’s not present in all the MQTT control packets. The payload of the message is limited to <strong>268,435,456</strong> bytes.</p><p>The length of the actual topic string is at most <strong>65536</strong> bytes. This is a limit imposed by the MQTT spec. It is also worth noting that the topic is encoded in UTF-8, so there might be less than <strong>65536</strong> characters available.</p><p>The MQTT spec details can be found <a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718018">here</a>.</p><h4>Connection loss awareness</h4><p>The broker is always aware of all the connections that clients establish or terminate. There are also certain features that utilize this event.</p><ul><li><a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901060">Last Will and Testament </a>(LWT), where every connected client describes his will, that upon disconnection a message will be published on a specific topic.</li><li><a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349231">Persistent Sessions</a> that we referred to earlier</li></ul><h4>Simple but not limiting authentication</h4><p>By design, MQTT offers a very <strong>simple authentication mechanism</strong>.</p><p>The <strong>CONNECT</strong> Packet contains <strong>Username</strong> and <strong>Password</strong> fields.</p><p><strong>Implementations can choose how to make use of the content of these fields.</strong> They may provide their own authentication mechanism, use an external authentication system such as LDAP <a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#RFC4511">[RFC4511]</a> or OAuth <a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#RFC6749">[RFC6749]</a> tokens, or leverage operating system authentication mechanisms. Implementations passing authentication data in clear text, obfuscating such data elements or requiring no authentication data should be aware, that this can give rise to Man-in-the-Middle and replay attacks.</p><p>TLS <a href="http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#RFC5246">[RFC5246]</a> can provide encryption of data sent over the network. There are valid TLS cipher suites that include a NULL encryption algorithm that does not encrypt data. To ensure privacy, Clients and Servers should avoid these cipher suites.</p><p>An application might independently encrypt the contents of its Application Messages. This could provide privacy of the Application Message both over the network and at rest. This would not provide privacy for other properties of the Application Message such as Topic Name.</p><p>Client and Server implementations can provide encrypted storage for data at rest such as Application Messages that are stored as part of a Session.</p><p>The use of VPNs to connect Clients and Servers can provide privacy of data across the section of the network covered by a VPN.</p><p>Every broker that exists in the market has its own modules to handle authentication but generally they can be extended further to support authentication mechanisms such as those mentioned above. These implementations are usually custom, although some brokers on the market have some modules already implemented.</p><h3>Best Practices</h3><p>Although there are no restrictions on how Apps using the protocol are implemented, there are some practices that are widely adopted.</p><h4>Topic structure</h4><ul><li><strong>Topics are meant to categorize information into groups,</strong> in terms of interest. The topic structure should resemble a tree structure and should make possible the utilization of wildcards for subscriptions. Information regarding a specific topic should provide the ability to be retrieved by multiple entities. To achieve that, <strong>consistency between topic layers should be maintained.</strong></li><li><strong>Do not use a leading forward slash </strong>since it introduces another unused topic layer in the tree with a zero character that is not used. This can cause unnecessary confusion.</li><li><strong>Never use spaces in a topic</strong>, UTF-8 has many different white space types and can cause trouble in debugging. Characters like these should be always avoided.</li><li><strong>Never subscribe to “#”</strong>, the client might not be able to process the load and might also put some unnecessary strain on the broker. If you need, for some reason let’s say store all messages on a database, you should implement some kind of plugin on the broker to achieve this.</li><li><strong>Specific topics and Extensibility should always be on the designer&#39;s mind since your topic structure should be able to extend.</strong> Topic name conventions should be maintained across all topics in order for your system to be able to adapt to new topic addition in the future without breaking the tree structure.</li><li><strong>Use a unique identifier on topics that refer to specific entities. </strong>You should be able to distinguish each entity’s information or group it by a single level wildcard.</li></ul><h4>Proper Service Levels</h4><p>As we already mentioned, <strong>service levels can be different from client to broker &amp; broker to client.</strong> Always take note that QOS2 service level is expensive and should be used only when it is absolutely necessary. QOS0 should be used only on non-sensitive data as message loss can occur. QOS1 should be used when receiving duplicate data will not cause any trouble.</p><h4>Connection Utilization</h4><p>In MQTT, <strong>establishing a connection is not a cheap task</strong>. Always avoid connections that will be utilized for a single information transaction. Maintain connection or send data to batches if possible. Single transaction connections almost ruin all the pros of the fast and compound nature of the protocol.</p><h4>Security</h4><p><strong>Security should never be taken lightly and TLS over TCP should always be used along with all the best practices that come along</strong>. Your connected entities will most likely be on a remote location meaning that your broker should be visible to the world. If your entities can execute commands or take orders from messages, then it’s not only a matter of security but also a matter of safety since sophisticated attackers can perform man-in-the-middle attacks at least.</p><p>This is a matter of great importance, not only on MQTT related implementations but, generally on IoT technologies, we all can remember what the <a href="https://en.wikipedia.org/wiki/Mirai_(malware)">Mirai-bot</a> attacks taught us in 2016.</p><h3>Brokers on the market</h3><p>There are several brokers out there that implement the MQTT protocol. Here are some of them that are my personal favorites.</p><p><strong>VerneMQ</strong> — Is particularly interesting due to it’s scaling capabilities and provided MQTT features.</p><p><strong>KubeEdge</strong> — (Mosquitto on Kubernetes) — having a broker “natively” on kubernetes.</p><p><strong>Mosca-Mosquitto </strong>— I actually started out experimenting some years ago using Mosca. Mosca and Mosquitto are most likely the best brokers to learn, test and debug.</p><h3>Conclusions</h3><p><strong>MQTT is a PUB/SUB communication protocol that enables light, fast and effective communication between entities.</strong> It has many features such as the multiple service levels, persistent sessions, last will &amp; testament and many more. It implements a unique tree topic structure that enables flexible management of information. Last but not least, there are several brokers and clients out there can cope with the scaling demands of the present.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8155f7475dd5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>