<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ng Tze Yang on Medium]]></title>
        <description><![CDATA[Stories by Ng Tze Yang on Medium]]></description>
        <link>https://medium.com/@ngtzeyang94?source=rss-3d1267537130------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*n11h7xpKmatAp7iECAUzDA.png</url>
            <title>Stories by Ng Tze Yang on Medium</title>
            <link>https://medium.com/@ngtzeyang94?source=rss-3d1267537130------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:36:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ngtzeyang94/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Go with Examples: Protobuf Encoding Mechanics]]></title>
            <link>https://ngtzeyang94.medium.com/go-with-examples-protobuf-encoding-mechanics-54ceff48ebaa?source=rss-3d1267537130------2</link>
            <guid isPermaLink="false">https://medium.com/p/54ceff48ebaa</guid>
            <category><![CDATA[distri]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Ng Tze Yang]]></dc:creator>
            <pubDate>Mon, 02 Nov 2020 02:14:40 GMT</pubDate>
            <atom:updated>2020-11-02T02:14:40.303Z</atom:updated>
            <content:encoded><![CDATA[<p>TLDR;</p><ul><li>Minimise usage of field numbers &gt; 15</li><li>Use packed option for long repeated fields</li><li>Embedded Messages are wrapped with metadata of higher level messages</li></ul><h3>Background</h3><p>Protobuffers is an encoding mechanism for message sending between servers &amp; services. It is highly popular at many companies mainly because of its great performance in terms of</p><p>(1) encoding &amp; decoding speed &amp;</p><p>(2) its ability to save bandwidth by being able encode into binary.</p><p>This blog will describe the mechanisms of how objects gets encoded to be sent over the wire. Follow along with the <a href="https://github.com/tng016/go_with_examples/tree/master/proto_encoding">examples</a> on your own!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xUEGUrR9gmj4dz_G.png" /></figure><h3>Part 1 — Message Structure</h3><p>Let’s look the simplest message me can send:</p><pre>//example 1 proto file</pre><pre>message SingleInteger {<br>    optional uint64 integer = 1;<br>}</pre><pre>//main.go</pre><pre>m1 := &amp;integers.SingleInteger{<br>    Integer: proto.Uint64(1),<br>}</pre><p>After Marshalling the above object, you will get the following []byte. For every object being marshalled, we will get some bytes encoding the metadata (data about the data) as well as the data itself to be sent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cvsX82NAbiFzsMlJT5xoIQ.png" /></figure><h3>MetaData</h3><p>Let’s first look at the metadata. There are two important metadata that needs to be communicated — it’s field number &amp; its type</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*plLEkHIE_H1mbZphyf5qmg.png" /></figure><p>Looking at the table provided provided on their <a href="https://developers.google.com/protocol-buffers/docs/encoding">website</a>, it is quite clear to see that the meta data is telling us that the data is of <strong>type 0 (Varint) </strong>and has <strong>field number of 1</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kP08qBEupH2gs7MSZcWAnw.png" /><figcaption>source: <a href="https://developers.google.com/protocol-buffers/docs/encoding">https://developers.google.com/protocol-buffers/docs/encoding</a></figcaption></figure><h3>Q: What is MSB &amp; what happens if we want to store a data value &gt; 127?</h3><h3>Part 2 — Variable length Integers (Varints)</h3><p>Let’s rewrite our example with a higher value:</p><pre>//example 2.1 proto file</pre><pre>message SingleInteger {<br>    optional uint64 integerA = 1;<br>}</pre><pre>//main.go</pre><pre>m1 := &amp;integers.SingleInteger{<br>    IntegerA: proto.Uint64(18789),<br>}</pre><p>Marshalling the object, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*rdNrVYRy8Z33t8XqnOs1tw.png" /></figure><p>To store value 18789, it will overflow the 7 bits allocated for data storage &amp; will require another 2 bytes to encode its data. This is where the MSB comes into play.</p><blockquote>Each byte in a varint, except the last byte, has the most significant bit (msb) set — this indicates that there are further bytes to come.</blockquote><p>So to get the data value, what you do is:</p><p>Step 1 — Remove the MSB for each byte</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*ghYDzbmrvJwYB1GiEsPDIA.png" /></figure><p>Step 2 — Rearrange them with the <strong>least significant byte first</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*EQ4kZFjDR-55NywGGCaLQA.png" /></figure><p>Step 3 — Join them Together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*KTc-7JFrDS3EF-vG0ko6Zw.png" /></figure><p>Calculating binary 100100101100101 → decimal 18789</p><h3>Q: Does this apply for the field number too?</h3><p>Yes! Let’s first look at Integer A which is on <strong>field number 15 </strong>and fits nicely into the four bits allocated to it within the MetaData byte.</p><p>But for Integer B, its <strong>field number 16</strong> will overflow the 4 bits allocated &amp; will require another byte to encode its metadata. This is where the MSB comes into play.</p><pre>//example 2.2 proto file</pre><pre>message SingleInteger {<br>    optional uint64 integer = 16;<br>}</pre><pre>//main.go</pre><pre>m1 := &amp;integers.SingleInteger{<br>    Integer: proto.Uint64(1),<br>}</pre><p>After applying the same steps above, you should be able to get the following bits to determine the field number = 16.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MjyY63HblJFyrxuZdIeFbQ.png" /></figure><h3>Q: How can we use this knowledge to our advantage?</h3><p>If you really want to be efficient, fields 1–15 should be be reserved for the <strong>most frequently used fields</strong>. Alternatively, using embedded messages where it makes sense is also another way to go. (see below)</p><h3>Part 3 — Signed Integers</h3><p>Looking at the variable integer types (Type 0) that protobuf supports, one interesting comparison is that between standard integers (int32, int64) and signed integer types (sint32, sint64).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*wvgtzk1aFlaIymOUvvY2pg.png" /></figure><blockquote>If you use int32 or int64 as the type for a negative number, the resulting varint is always ten bytes long…(but) If you use one of the signed types, the resulting varint uses ZigZag encoding, which is much more efficient.</blockquote><p>Let’s take a look at an example:</p><pre>//example 3 proto file</pre><pre>message SignedInteger {<br>    optional int32 integer = 1;<br>    optional sint32 s_integer = 2;<br>}</pre><pre>//main.go</pre><pre>m1 := &amp;integers.SignedInteger{<br>    Integer:  proto.Int32(-1),<br>    SInteger: proto.Int32(-1),<br>}</pre><p>Here’s the Encoding for the <strong>Integer</strong> field</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*V0dAPqkfzT-mkGaoaVk7CQ.png" /></figure><p>And here’s the Encoding for the <strong>SInteger</strong> field</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*k9Q9wuZyjgUo-L7sB7MZmQ.png" /></figure><h3>Q: Why 10 bytes?</h3><p>Protobuf stores a negative int32 in 64 bits (I assume because to allow a negative int32 to be able to be unmarshalled into a int64 field).</p><p>Remember each data byte only has 7 useful bits for storing data. So to store 64 bits, we will need 10 bytes.</p><h3>Q: What is ZigZag encoding? Why is the data byte for SInteger = 1?</h3><p>Google probably noticed that for most applications, the usual integers to be used in messages are usually close to zero. So they used the following encoding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGJOq6D6LCNADKR-j9cjjQ.png" /><figcaption>source: <a href="https://developers.google.com/protocol-buffers/docs/encoding">https://developers.google.com/protocol-buffers/docs/encoding</a></figcaption></figure><p>This way, negative integers need not waste space storing the entire two’s complement in 10 bytes when it can be done in less.</p><h3>Part 4 — Strings</h3><p>The principle design consideration for strings is that unlike ints, it can be of variable lengths from 1 byte to infinitely long.</p><p>Therefore what we do is to insert the length of the string into the metadata.</p><pre>//example 4 proto file</pre><pre>message ProtoString {<br>    optional string s = 1;<br>}</pre><pre>//main.go</pre><pre>m1 := &amp;protostrings.ProtoString{<br>    S: proto.String(&quot;123&quot;),<br>}</pre><p><strong>After marshalling…</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*-2Hr7_D0HMKEEvNHrfHzdQ.png" /></figure><p><strong>Some things to take note of…</strong></p><p>1) The Metadata value is 10 now instead of 8 now that we are using a type 2 value</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*Bs6c_2c3BKvAd8CL_OzCWQ.png" /></figure><p>2) Length byte is 6 as we have 6 following bytes encoding the string data</p><p>3) The following byte thereafter stores the UTF encoding of the string.</p><h3>Part 5 — Repeated fields</h3><p>For repeated fields, the encoding mechanism varies slightly between proto2 and proto3 with thee use of packed fields. Let’s first start with the use of packing</p><pre>//example 5 proto file</pre><pre>message RepeatedInteger {<br>    repeated uint64 integer = 1;<br>    repeated uint64 packed_integer = 2 [packed=<strong>true</strong>];<br>}</pre><pre>//main.go</pre><pre>m1 := &amp;repeated.RepeatedInteger{<br>    Integer:       []uint64{1, 2, 3},<br>    PackedInteger: []uint64{1, 2, 3},<br>}</pre><p><strong>Encoding for Integer Field</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*TyxUqofitm64sdnRgfNrMg.png" /></figure><p><strong>Encoding for Packed Integer Field</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*XSB-oCvT2BHpjnLayeiMTg.png" /></figure><p>As we can see, unpacked repeated fields repeat the metadata byte of each element within the array while with the packing option turned on, all data bytes share the same metadata.</p><p>The encoding mechanism varies slightly between proto2 and proto3 with thee use of packed fields. In proto2, the packing option must be explicitly toggled in the example above, but for proto 3 packing will be used by default.</p><h3>Part 6 — Embedded messages</h3><p>In our final example, let’s see how messages within messages are encoded.</p><pre>//example 5 proto file</pre><pre>message TopMessage{<br>    optional EmbeddedMessage m = 1;<br>}</pre><pre>message EmbeddedMessage{<br>    optional uint64 i = 1;<br>    optional string s = 2;<br>    optional bool b = 3;<br>}</pre><pre>//main.go</pre><pre>m1 := &amp;embedded.EmbeddedMessage{<br>    I: proto.Uint64(1),<br>    S: proto.String(&quot;123&quot;),<br>    B: proto.Bool(<strong>true</strong>),<br>}</pre><pre>m2 := &amp;embedded.TopMessage{<br>    M: m1,<br>}</pre><p><strong>Result:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wikNSHgc518OVmUz3bacFA.png" /></figure><p>If you have noticed, this is very similar to TCP/HTTP headers where lower level messages are wrapped with a header from the higher level messages. Let’s see if you correctly identify why each byte is encoded as such. I’ve already gave you half the answer and told you what the byte encodes for</p><h3>Part 7 — Conclusion &amp; Further Puzzles</h3><p>Protobuf is a very neat &amp; efficient encoding mechanism for message sending and on top of the that, the code is so intuitive as you can see by the simple examples.</p><p>There are many more features to protobuf that was not mentioned in this tutorial. But I hope i’ve provided the basics for you to carry on by yourself.</p><p>If you want, there are some interesting cases I have listed below. How many do you think you can guess correctly?</p><ul><li>Marshalling into a different object with same field types but different names?</li><li>Marshalling into a different object with different field types?</li><li>How will protobuf encode Packed strings?</li><li>Can you unmarshall a packed repeated []uint64 into an embedded message &amp; vice versa?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54ceff48ebaa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The curse of the highly intelligent, highly educated and highly successful]]></title>
            <link>https://ngtzeyang94.medium.com/the-curse-of-the-highly-intelligent-highly-educated-and-highly-successful-541c0566bc0e?source=rss-3d1267537130------2</link>
            <guid isPermaLink="false">https://medium.com/p/541c0566bc0e</guid>
            <category><![CDATA[reflections]]></category>
            <category><![CDATA[current-affairs]]></category>
            <category><![CDATA[thoughts]]></category>
            <category><![CDATA[singapore]]></category>
            <category><![CDATA[humility]]></category>
            <dc:creator><![CDATA[Ng Tze Yang]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 03:38:56 GMT</pubDate>
            <atom:updated>2020-08-21T03:38:56.695Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unheeded warnings from a distant age</h3><p>8 years ago while in high school, a wise man gave a speech at our founders’ day celebration. His name was Bilahari Kaushikan and he was then the Permanent Secretary of Foreign Affairs. Re-reading his speech from all those years ago brings out many lessons in humility and introspection. I sincerely beseech you to spend the time to contemplate on this magnificent piece of writing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*FKJwE3u6TdOF0q8oHRnkWw.jpeg" /><figcaption>Raffles Institution — my Alma Mater and that of Bilahari Kaushikan (Source: The Straits Times)</figcaption></figure><h3>The Speech:</h3><p>Eighty-five years ago an American writer by the name of Thornton Wilder published a short novel entitled The Bridge of San Luis Rey. The book has never been out of print, but deserves to be better known.</p><p>The novel begins at noon on a certain day in 1714 when a bridge in Peru — “the finest bridge in all Peru”, writes Wilder — inexplicably collapses and five people who happen at that moment to be crossing, plummet to their deaths.</p><p>The tragedy is witnessed by a devout Franciscan monk, in Peru for missionary work among the natives, who immediately asks himself “Why did this happen to those five?”</p><p>The monk is convinced that it was not a random event but some manifestation of God’s Will for some greater end and vows to investigate to so as to prove to the natives the necessity of divine purpose. But his investigation runs afoul of the Inquisition and he is burnt at the stake.</p><p>Wilder poses, but never directly answers, the question: “Is there a direction and meaning in lives beyond the individual’s own will?” The point, of course, is that it could have been anyone of us on that metaphorical bridge.</p><p>I do not think that there is any particular meaning, pattern or direction, divine or secular, in the drift of human events. History, as Winston Churchill is reported to have remarked, is just one damned thing after another. The innocent die young and the wicked flourish; and not necessarily in equal measure either because to the wicked the innocent are often prey.</p><h3>The world is far too complex a place to be comprehended in any holistic way by the human mind.</h3><p>It is made up of too many moving parts interacting in too many unpredictable ways for human reason to grasp.</p><p>I mean, of course, the social world: the world of human interactions, human relationships and human institutions; of love and hatred, politics and economics, war and peace, infused with emotions like anger, pity, joy and sorrow, and not the material world of rocks and stones and trees and the earth’s diurnal course.</p><p>In the material world, the apple will always fall whether or not Newton was there to observe it. In the material world, all phenomena must ultimately conform to the laws of physics. In the material world, when we return to earth and ashes, we too will confirm to the laws of physics.</p><p>But in the meantime we inhabit a social world of sentient beings who observe, think and respond so that our every effort to act or comprehend alters what we try to comprehend and every thought and action begets a never ending, ever shifting kaleidoscope of unpredictable possibilities that makes all social science an oxymoron.</p><blockquote><strong>Reason may distinguish man from beast, but the sum of the interactions of different reasons; of many logics, is only coincidentally and occasionally logical. </strong>That is why actions always have unintended consequences even if they are not always immediately apparent, and our best laid plans and most fervent hopes are constantly ambushed by chance and events.</blockquote><p>Most things eventually fail. The shade of Ozymandias hovers unseen but omnipresent over every human enterprise, biding its time.</p><p>The ancient Greeks advised us to call no man happy until he was dead. This is good advice. We can be reasonably certain of something only after it has occurred. The only true knowledge is historical, and even then there is always room for argument over interpretation. None of us ever sees or understands the same thing, no matter how conscientiously we try to observe or communicate.</p><p>As I stand here speaking to you, at least three different things are occurring simultaneously: first, what I think; second, what I say to convey what I think which, whether because of the limitations of language or by design, will not always be the same as what I think: deception and self-deception are intrinsic parts of human nature; and third what you hear and understand of what I had intended to convey which is again not necessarily the same thing.</p><p>One could call this, after the title of a short story by the Japanese author Ryunosuke Akutagawa, the Rashomon phenomenon. It makes for a world without fixed meaning, which accentuates its fundamental incomprehensibility. A world in which the past can only be partially known, the present is largely unknown and the future certainly unknowable.</p><p>None of us asked to be born. Yet having had life thrust upon us, we must, unless bent on suicide, nevertheless live. Although we can only, if dimly and darkly, know backwards, we have to live forwards.</p><p>No one can live in a constant Hamlet-like state of existential doubt. We must profess a certainty that we do not necessarily feel. To keep the metaphysical horror of unfathomable meaninglessness at bay, we all, singly or collectively, consciously or unconsciously, adopt mental frameworks to simplify a complex reality in order to deal with it.</p><blockquote><strong>But the fundamental mode of thought that underpins these trappings of reason is still theological in that whether our belief is in Reason or in God</strong></blockquote><p>Since the Enlightenment of the 17th Century, belief in Reason has replaced belief in God as the primary organizing mental framework of society. We are all the creatures of this western defined modernity and the most successful of the non-western countries, Singapore among them, are precisely those who have embraced it the most closely.</p><p>Reason’s children include law and justice, philosophy, literature and the arts, economics and other social sciences and even the very belief in reason, progress, technology and science. But the fundamental mode of thought that underpins these trappings of reason is still theological in that whether our belief is in Reason or in God, it is still mere belief and not epistemologically provable beyond all doubt. There is no end to philosophy any more than there can be an end to history.</p><p>Stated in another way, none of Reason’s children have an autonomous reality separate from our apprehensions of them. They are socially constructed artefacts; frameworks of ideas that we have chosen to believe in, in order to comprehend the world and comprehend in order live in a particular way.</p><p>Their utility is thus purely instrumental. They are at best all only partially and contingently right which means, of course, that they are all also always at least partially wrong. That includes, by the way, the ideas I am presently expounding.</p><p>I advance these arguments not to instil cynicism or despair but to suggest the possibility of liberation and hope.</p><h3>A rock is forever only a rock.</h3><p>But human beings are defined by their potentialities, and since there is no predetermined meaning to the unfolding of events, the potentialities are equally boundless. Were it not so, Singapore should not exist as a sovereign and independent country.</p><p>The only meaning in life that can exist is that which we create for ourselves. And unless we want our lives to be merely a slow, selfish dying, we ought to try to create some meaning larger than ourselves.</p><p>This is, to my mind, an absolute duty imposed by the human condition, even if we know that uncertainty and error are constants and that we are always writing on sand before the advancing tide. Our duties to our families, our friends and our country endure when even hope is dead.</p><p>I am sure that by now many of you are harbouring a thought that you are too well brought up to speak out loud: this idiot exaggerates.</p><p>Of course, I exaggerate. But only a little, and only for clarity’s sake and not to distort or mislead. So let me restate my essential point in a different way.</p><blockquote><strong>… it is but a tiny step to being convinced that anyone who does not share your conviction is not just ignorant but evil</strong></blockquote><p>Do not confuse the depth of sincerity with which you or others hold an idea, or the number of people who sincerely hold an idea, with its validity. Sincerity is an over-rated virtue, if indeed it is a virtue.</p><p>All of you may be suddenly seized with the sincere conviction that that pigs should fly. But pigs will nevertheless never sprout wings no matter how devoutly you hope for them to escape the surly bonds of earth.</p><p>And if you, ignoring the possibility of error, sincerely believe that pigs ought to fly; or that God’s Will has been revealed to you; or that you are one of the elect to whom the direction of History’s cunning passages has been vouchsafed, then it is but a tiny step to being convinced that anyone who does not share your conviction is not just ignorant but evil. Then for the greater glory of PIGS or HISTORY or GOD, all spelt in capital letters, it is only a tinier further step to seeing it as your bounden DUTY, again spelt with capitals, to expunge the evil.</p><p>And it all inevitably ends as Wilder’s poor monk did, in flames at the stake.</p><p>Rather than sincerity, if we want to do some trifling and ephemeral good or at least to minimize harm, we should approach life with an ironic and humane scepticism.</p><p>Irony to ensure that we retain a sense of proportion and as ballast against the inevitability of unintended consequences: today’s error being the correction of yesterday’s error. Humanity so that we may empathize with logics other than our own, if only to better manoeuvre to impose our will because in a world of competing logics, if we hope to do any good, we cannot hope to do so by logic alone. And scepticism because the possibility of deception, our own self-deceptions if not those of others, casts constant shadows over every human action.</p><h3><strong>The curse of the highly intelligent, highly educated and highly successful</strong></h3><p>I have chosen to dwell on this at what you may consider inordinate length, because Raffles Institution likes to consider itself unique. Ladies and Gentlemen, I am sorry to inform you that RI is no longer unique.</p><p>You are now only one of a number of similar elite educational institutions from which will come a disproportionate number of scholarship recipients and a disproportionate number of leaders in the civil service, the professions, business, the Arts and the academy. And all these institutions are united by a certain sense of entitlement, possibly so profound as to be quite unconscious.</p><p>I do not blame you for this. All of you are highly intelligent. You will be very well educated. And the odds are that you will be more than averagely successful in your careers.</p><blockquote><strong>But understand that you will therefore also be more vulnerable to the curse of the highly intelligent, highly educated and highly successful: this curse is the illusion of certainty; the conviction of the omnipotence of your ideas.</strong></blockquote><p>This is the delusion that your ideas or words are validated by mere virtue by being thought or uttered by you! YOU and not some lesser being. And the more intelligent and the more successful and the more highly educated, the deeper the delusion. “The learned”, Adam Smith is reported to have said, “ignore the evidence of their senses to preserve the coherence of the ideas of their imagination.”</p><p>Shortly after the 2008–2009 financial crisis, Alan Greenspan, the former Chairman of the US Federal Reserve, a powerful and erudite man, confessed in testimony to a Senate. Hearing that his intellectual assumptions of a lifetime had been shaken and he was still trying to understand what happened. I do not know if he has since come to any conclusions. But it was clear that prior to the near global disaster, he had never even faintly contemplated the possibility that his beliefs may have been in error. We are all still paying the price for his certainties.</p><h3>Yours will be a generation that that will live through times of more than usual uncertainty.</h3><p>A global transition of power and ideas is underway. Transition to what, no one can yet say. We have no maps and will have to improvise our way forward the best we can. It will be a transition measured in decades and not just a few years, and it is your misfortune that it is occurring as the technology of the internet is making us solipsistic.</p><p>The internet conflates and confuses our opinion with information and tempts us to immerse ourselves only in a circle of those who share and reinforce our own interests and views. It shortens attention spans and privileges the new and novel over any notion of lasting value. Social media like Facebook have perverted the common meaning of ‘friend’ and ‘like’ beyond all recognition. Only a solipsist or, what is much the same thing, a narcissist, would think that what he or she had for lunch would be of wider interest; and only those with vacuous minds would be interested. And this at a time when the safe navigation of uncharted waters requires a prudent modesty, openness and some minimal capacity for sustained thought.</p><blockquote>And yet the internet and its associated technologies is indispensable to modern life. We need it to prosper. But what its ultimate effects will be on society, on governance, on international relations, on the very way we think, no one yet knows.</blockquote><p>I certainly have no answers. As you, the anointed ones, ready yourselves to assume authority and responsibility under these challenging circumstances. I can do no more than to remind you of what Sir Olivier Cromwell wrote to the Synod of the Church of Scotland in 1650. He was trying to persuade the Scots not to embrace the Royalist cause of King Charles the Second and so avert civil war.</p><p>Gentlemen, he wrote, “I beseech you, in the bowels of Christ” — and I should explain that in the 17th Century the bowels were considered to be the seat of pity or the gentler emotions — Gentlemen, Cromwell wrote, “I beseech you, in the bowels of Christ, think it possible that you may be mistaken”.</p><p>So, Ladies and Gentlemen of the 21st Century, I too beseech you from whatever portion of anatomy you consider most dear, think it possible that you may be mistaken.</p><p>Before I conclude, you may wish to know how it all ended.</p><p>Cromwell’s advice was not heeded. Shortly thereafter, the third English Civil War broke out. This set in motion a historical trajectory of political, social and economic changes that led to modern Britain, the industrial revolution, the East India Company, Sir Stamford Raffles, the British Empire, the founding of Singapore and ultimately, you and I.</p><p>And all because good advice fell on deaf years.</p><p>What better way to appreciate the irony and contingency of events than to ponder what may have happened if Cromwell’s advice was in fact taken and civil war avoided. And as you do so, consider also the possibility that you may be mistaken when you think you are mistaken.</p><p>And with that final paradox I will end.</p><p><strong><em>Bilahari Kausikan was the Permanent Secretary of Foreign Affairs at the time of writing.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=541c0566bc0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go With Examples: Keeping Your Modules Compatible]]></title>
            <link>https://medium.com/swlh/go-with-examples-keeping-your-modules-compatible-2bab5a5b9022?source=rss-3d1267537130------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bab5a5b9022</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[golang-development]]></category>
            <dc:creator><![CDATA[Ng Tze Yang]]></dc:creator>
            <pubDate>Sun, 09 Aug 2020 11:56:20 GMT</pubDate>
            <atom:updated>2020-11-02T14:35:16.877Z</atom:updated>
            <content:encoded><![CDATA[<h3>TLDR;</h3><ul><li>When extending a function, NEVER change the function signature</li><li>Make sure you have thought about the future uses before exposing an Interface to public, because you’re NEVER gonna change that</li><li>There’s Backward Compatibility (code can compile) and Behaviour Compatibility (code acts the same way) to consider</li></ul><h3>Background</h3><p>In this world, we have two kinds of repositories:</p><ol><li>Application Repositories — repositories that is only used &amp; seen by you and your team that contains the business logic of your App</li><li>Shared Libraries — Repositories that are used by many other application repositories (think any standard library, SQL package, Redis, any open source pkg etc…)</li></ol><p>Because many application repositories rely so heavily on Shared Libraries, we need to make extra care when writing them to 1) maintain readability and most importantly, 2) ensure backward compatibility.</p><p>This blog post hopes to provide junior devs with an illustration of the concepts taught in the Golang blog’s latest post: <a href="https://blog.golang.org/module-compatibility">https://blog.golang.org/module-compatibility</a>. I will first show simple examples of the patterns discussed and also point to some examples already illustrating these points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9c1zBCnaw1cQJlAbuyLECA.jpeg" /><figcaption>Isn’t she Go-regeous?</figcaption></figure><h3>Part 1: Extending Functions</h3><p>Business requirements always change. More often than not, requirements are added and increase complexity to our code.</p><p>How can we add more parameters to existing functions without breaking backwards compatibility for callers already using the function?</p><h3>Simple Example</h3><p>Let’s say you have a coffee machine that makes plain black coffee.</p><pre>func MakeCoffee(){<br>   fmt.Println(&quot;coffee!&quot;)<br>}</pre><p>How can we extend the function to add sugar to our coffee?</p><h3>Bad — adding variadic arguments</h3><pre><a href="https://play.golang.org/p/clVdxH5N78E">https://play.golang.org/p/clVdxH5N78E</a></pre><pre>func MakeCoffee(<strong>withSugar</strong> ...bool){<br>   if <strong>withSugar</strong> != nil &amp;&amp; <strong>withSugar</strong>[0]{<br>      fmt.Println(&quot;coffee with sugar!&quot;)<br>   } else {<br>      fmt.Println(&quot;coffee!&quot;)<br>   }<br>}</pre><p>While this simple example may work in practice, there are many reasons why we should not do this:</p><ol><li>It is only extensible once, we cannot extend it again with another variadic argument.</li><li>Call compatibility != backward compatibility. By changing the function signature we will always break backward compatibility shown here:</li></ol><pre><a href="https://play.golang.org/p/WAK_RL4Ze0v">https://play.golang.org/p/WAK_RL4Ze0v</a></pre><pre>func main(){<br>   var coffeeFunc func() = MakeCoffee //compile err<br>   coffeeFunc()<br>}</pre><h3>Good — create a new function</h3><p>A better idea is to create a new function to implement the more complex logic. We can then change the implementation of the first function with the default <em>false</em> option.</p><pre><a href="https://play.golang.org/p/55zmgeaoolF">https://play.golang.org/p/55zmgeaoolF</a></pre><pre>func MakeGoodCoffee(<strong>withSugar</strong> bool) {<br>   if <strong>withSugar</strong> {<br>      fmt.Println(&quot;coffee with sugar!&quot;)<br>   } else {<br>      fmt.Println(&quot;coffee!&quot;)<br>   }<br>}</pre><pre>func MakeCoffee(){<br>   MakeGoodCoffee(<strong>false</strong>)<br>}</pre><h3>Better — plan for extension from the start</h3><p>As a developer, if you know that more complicated logic may come in the future, it would be useful to design your functions to accommodate the changes easily with the following examples</p><h3>(a) Use a config struct</h3><pre><a href="https://play.golang.org/p/7NnZHUE16Bx">https://play.golang.org/p/7NnZHUE16Bx</a></pre><pre>type CoffeeConfig struct{<br>   WithSugar bool<br>}</pre><pre>func MakeCoffee(<strong>config</strong> *CoffeeConfig){<br>   if <strong>config</strong> != nil &amp;&amp; <strong>config</strong>.WithSugar{<br>      fmt.Println(&quot;coffee with sugar!&quot;)<br>      <strong>return<br>   </strong>}<br>   fmt.Println(&quot;coffee!&quot;)<br>}</pre><p>With a config struct, you can continually add options for more complicated logic into the config without breaking backward compatibility.</p><p>A problem with this is that we have to consider the default option if a nil object is passed into function (see <a href="https://play.golang.org/p/7NnZHUE16Bx">playground</a>).</p><h3>But Wait!</h3><p>There is an edge case where changing the config struct <strong>will break compatibility</strong>, if we are performing comparisons on structs. Here’s what Golang says:</p><blockquote><em>Struct values are comparable if all their fields are comparable. Two struct values are equal if their corresponding non-</em><a href="https://golang.org/ref/spec#Blank_identifier"><em>blank</em></a><em> fields are equal.</em></blockquote><blockquote><em>- </em><a href="https://golang.org/ref/spec#Comparison_operators"><em>Golang Reference Specification</em></a></blockquote><p>In other words, if you add on a <strong>slice</strong>, <strong>map</strong> or <strong>function</strong> field into a struct, we will not be able to perform a comparison on the struct. Here’s an example:</p><pre><a href="https://play.golang.org/p/j533RPDkOSo">https://play.golang.org/p/j533RPDkOSo</a></pre><pre>//OK!<br>type ComparableStruct struct {<br>   i <strong>int<br>   </strong>s string<br>   b bool<br>}</pre><pre>//Unable to perform == or != operation on this struct<br>type UncomparableStruct struct {<br>   i <strong>int<br>   </strong>s string<br>   b bool<br>   v []<strong>int<br></strong>}</pre><p>This is a rarer case but it is still important to keep in mind!</p><h3>(b) Make config struct a method receiver</h3><pre><a href="https://play.golang.org/p/W5TepucGkm0">https://play.golang.org/p/W5TepucGkm0</a></pre><pre>type BeverageMaker <strong>interface</strong>{<br>   MakeBeverage()<br>}</pre><pre>type CoffeeOrder struct{<br>   WithSugar bool<br>}</pre><pre>func (order *CoffeeOrder) MakeBeverage(){<br>   <strong>if</strong> order.WithSugar{<br>      fmt.Println(&quot;coffee with sugar!&quot;)<br>      <strong>return<br>   </strong>}<br>   fmt.Println(&quot;coffee!&quot;)<br>}</pre><p>Unlike (a), this allows more flexibility in the larger application context if we are dealing with interfaces (in this case, DrinkDispenser) but at the cost of more difficult readability. Both (a) and (b) are useful in their own specific situations.</p><h3>(c) Functional options pattern</h3><p>You could also use the functional options to extend your function too. Click on <a href="https://www.sohamkamani.com/golang/options-pattern/">this webpage</a> if you are unfamiliar with the pattern. Also while you are on your detour, search for<strong> decorator patterns</strong> too! It’s very similar to what we’ve been discussing thus far.</p><pre><a href="https://play.golang.org/p/8BFPMGpeo_Q">https://play.golang.org/p/8BFPMGpeo_Q</a></pre><pre>type CoffeeOptions func()string</pre><pre>func WithSugar()string{<br>   <strong>return</strong> &quot;with sugar&quot;<br>}</pre><pre>func MakeBeverage(options ...CoffeeOptions){<br>   coffee := &quot;coffee&quot;<br>   <strong>for</strong> _,opt := range options{<br>      coffee += opt()<br>   }<br>   fmt.Println(coffee)<br>}</pre><p>Problem with this is that with many options to specify, your function call grows really long like this:</p><pre>Coffee := coffee.MakeBeverage(coffee.WithSugar(), coffee.WithIce(), coffee.WithCream(), coffee.WithMilk(), coffee.WithStraw(), …)</pre><p>It also adds more lines and complexity into your package. Not something you need for a simple cup of coffee.</p><h3>Part 2: Extending Interfaces</h3><p>Now on to extending interfaces. It should be obvious that adding a new method directly to an existing interface will break compatibility.</p><p>In our coffee example, let’s say we now want our coffee stirred:</p><pre><a href="https://play.golang.org/p/9pdm1CshkMn">https://play.golang.org/p/9pdm1CshkMn</a></pre><pre>type BeverageMaker <strong>interface</strong>{<br>   MakeBeverage()<br>   StirBeverage() //BAD<br>}</pre><pre>//no longer implements DrinkDispenser interface<br>type CoffeeShop struct{<br>   CoffeeMaker BeverageMaker<br>}</pre><pre>func (shop *CoffeeShop) Serve(){<br>   shop.CoffeeMaker.MakeBeverage()<br>}</pre><h3>OK — Create new interface, dynamically check against new interface</h3><p>We can fix the problem with the following steps:</p><ol><li>Create a new interface, <em>BeverageStirrer</em></li><li>In our original <em>Serve</em> function, dynamically check if our C<em>offeeMaker</em> can stir</li><li>If it can, stir the drink! else, never mind</li></ol><pre><a href="https://play.golang.org/p/uuHDA7vo8Ps">https://play.golang.org/p/uuHDA7vo8Ps</a></pre><pre>type BeverageMaker <strong>interface</strong>{<br>   MakeBeverage()<br>}</pre><pre>//New interface<br>type BeverageStirrer <strong>interface</strong>{<br>   StirBeverage()<br>}</pre><pre>type CoffeeShop struct{<br>   CoffeeMaker BeverageMaker<br>}</pre><pre>func (shop *CoffeeShop) Serve(){<br>   shop.CoffeeMaker.MakeBeverage()</pre><pre>   //only stir if your coffeeMaker has the ability to do so!<br>   <strong>if</strong> stirrer, ok := shop.CoffeeMaker.(BeverageStirrer); ok{<br>      stirrer.StirBeverage()<br>   }<br>}</pre><h3>Better — Don’t use an interface if you don’t need to</h3><p>It is very difficult to change interfaces once implemented. Unless you are confident in the future uses of your interfaces, it would be better to stick with concrete interfaces.</p><pre><a href="https://play.golang.org/p/xSvTRvFWjfq">https://play.golang.org/p/xSvTRvFWjfq</a></pre><pre>type SimpleCoffeeShop struct{<br>   CoffeeMaker SimpleCoffeeMaker<br>}</pre><pre>//this simple coffee maker only able to make beverages<br>func (shop *SimpleCoffeeMaker) Serve(){<br>   shop.CoffeeMaker.MakeBeverage()<br>}</pre><pre>//TODO: implement supporting function <br>//SimpleCoffeeMaker.MakeBeverage()</pre><pre>type LuxuryCoffeeShop struct{<br>   CoffeeMaker LuxuryCoffeeMaker<br>}</pre><pre>func (shop *LuxuryCoffeeShop) Serve(){<br>   shop.CoffeeMaker.MakeBeverage()<br>   shop.CoffeeMaker.StirBeverage()<br>}</pre><pre>//TODO: implement supporting function <br>//LuxuryCoffeeMaker.MakeBeverage() and<br>//LuxuryCoffeeMaker.StirBeverage()</pre><p>As you can see, the downside to this is that the code size increases to a larger extend than using interfaces. But on the good side, less complexity + more clarity!</p><h3>Even Better — force <a href="https://golang.org/doc/effective_go.html#embedding">embedding</a> with private methods</h3><p>If you have an interface that you don’t want others implementing, such as if you are still in early stages of developing your interface, you can add any private method to the interface. Doing so will force others to embed your own implementations and are unable to create their own implementations!</p><pre><a href="https://play.golang.org/p/1FKMJcvnix2">https://play.golang.org/p/1FKMJcvnix2</a></pre><pre>type BeverageMaker <strong>interface</strong>{<br>   MakeBeverage()<br>   <strong>private</strong>()<br>}</pre><pre>//BeverageMaker Embedded into CoffeeShop<br>type CoffeeShop struct{<br>   BeverageMaker<br>}</pre><pre>//Only CoffeeMaker allowed to be embedded into CoffeeShop<br>type CoffeeMaker struct{}</pre><pre>func (cm *CoffeeMaker) MakeBeverage(){<br>   fmt.Println(&quot;Make Coffee!&quot;)<br>}</pre><pre>func (cm *CoffeeMaker) <strong>private</strong>(){<br>   //nil fn<br>}</pre><p>Because the signature of <strong>coffee.private()</strong> cannot be replicated outside of the coffee package, users cannot write their own implementations of <strong>BeverageMaker. </strong>Thus they are forced to embed a the CoffeeMaker object into a CoffeeShop, or any other implementations you have written in the coffee package</p><h3>Behaviour Breaking Changes:</h3><p>In all of the above example we discussed backwards compatibility on your functions and interfaces (changes that cause code to fail compilation). Even if user code is able to compile, your new implementation may have behavioural breaking changes, when results of the function call do not align with the user’s previous expectation.</p><p>The Golang blog has a nice example of this. Typically, users expect <a href="https://pkg.go.dev/encoding/json?tab=doc#Decoder">json.Decoder</a> to ignore fields in the JSON that is not in the argument struct.</p><pre><a href="https://play.golang.org/p/CQx6XShpew-">https://play.golang.org/p/CQx6XShpew-</a></pre><pre><strong>const</strong> jsonStream = `<br>   {&quot;Name&quot;: &quot;Ed&quot;, &quot;Text&quot;: &quot;Knock knock.&quot;, &quot;Email&quot;: &quot;abc@gmail.com&quot;}</pre><pre>`</pre><pre>type Message struct {<br>   Name, Text string<br>}</pre><pre>...after decoding...</pre><pre>fmt.Printf(&quot;%+v&quot;, message) // {Name:Ed Text:Knock knock.}</pre><pre>//opt in to new logic<br>dec.DisallowUnknownFields()</pre><pre>fmt.Printf(&quot;%+v&quot;, message) // err: json: unknown field &quot;Email&quot;</pre><p>With new changes, they wanted to return an error if there were such fields. What they ended up doing was you had to opt-in to the new behaviour by calling <strong>DisallowUnknownFields(). </strong>Otherwise, behaviour will default back to the original.</p><h3>Conclusion:</h3><p>With more devs using your library, comes great responsibility. Be extra careful when writing code that other people use!</p><p>Do your best when thinking about <strong>Compatibility</strong>, <strong>Simplicity</strong> and <strong>Readability.</strong></p><p>Doing so will save us all a ton of effort and time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bab5a5b9022" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/go-with-examples-keeping-your-modules-compatible-2bab5a5b9022">Go With Examples: Keeping Your Modules Compatible</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Inconspicuous Codis/Redis cluster CrossSlot bug]]></title>
            <link>https://ngtzeyang94.medium.com/the-inconspicuous-codis-redis-cluster-crossslot-bug-b2b7a33bf9e9?source=rss-3d1267537130------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2b7a33bf9e9</guid>
            <category><![CDATA[bugs]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[redis-cluster]]></category>
            <dc:creator><![CDATA[Ng Tze Yang]]></dc:creator>
            <pubDate>Thu, 23 Jul 2020 13:19:31 GMT</pubDate>
            <atom:updated>2020-07-23T13:19:31.245Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background — Spot the Bug!</h3><p>A few months ago I was given a seemingly simple ticket. A downstream team wanted a simple API with the following logic</p><h4>Request: <strong>Given a list of userIDs</strong></h4><h4>Response: Find the number of conversations by these users with at least 1 unread message</h4><h4>We currently store the data of the unread messages in a redis sorted set with the following schema</h4><pre>╔══════════════════╦═════════════════════════╦════════════════╗<br>║        Key       ║          Value          ║      Type      ║<br>╠══════════════════╬═════════════════════════╬════════════════╣<br>║ user_id{<strong>USER_ID</strong>} ║ {<strong>convo_id1, convo_id2…</strong>} ║ Sorted Set (Z) ║<br>╚══════════════════╩═════════════════════════╩════════════════╝</pre><p>To answer the query, we can write the following LUA script to retrieve the necessary Data, by inputting the proper key slice into the script</p><pre><strong>GET_UNREAD_CONV_COUNT_LUA</strong> = `<br>   local t = {}<br>   for i, key in ipairs(KEYS) do<br>      table.insert(t, redis.call(&#39;ZCARD&#39;, key))<br>   end<br>   return t<br>`</pre><h3>There is currently a bug in the methodology described above. Can you find it?</h3><p>Hint: Under some circumstances, it will give a wrong answer but under other circumstance, it will work perfectly fine. Review some of the assumptions I had when developing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xlQY0Hyh925OXMOoZ_baww.png" /><figcaption>create your own GOpher avatar at <a href="https://gopherize.me/">https://gopherize.me/</a></figcaption></figure><h3>In the meantime, let’s discuss about some Distributed Redis Systems:</h3><h3>Codis</h3><p>Codis is an open-sourced distributed redis system that may be used for storing data that changes frequently and needs to be retrieved quickly &amp; frequently. It is very similar to Redis Clusters but I will use Codis to demonstrate the basics of these distributed redis systems as Codis provides a very pretty front-end to visualise what happens underneath.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fXI7StDRZ-9BWVmGwWX9Ng.png" /><figcaption>Architecture of Codis System. We will only be discussing the basics — Codis Groups &amp; Proxies</figcaption></figure><h3>1. Single Redis Instance</h3><p>Starting with the basic building block, I’m sure you are all familiar with a single redis instance. If not, here’s all you need to know:</p><ul><li>Redis implements a in-memory key-value storage. Every value stored on Redis has a corresponding key to access the data</li><li>It can perform CRUD (create, read, update, delete) operations on the stored data, either directly or via a <strong>LUA Script</strong>.</li></ul><p>However, having a single Redis to handle all requests have limitations on:</p><ol><li>Storage Size</li><li>Queries able to handle per second (QPS)</li></ol><p>A single instance is able to handle a signifiant amount of load but if we are talking about &gt; 100,000 QPS, we need a way to scale this out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*aBl4tU8DXFCaAfyzMooV7g.png" /><figcaption>there is only so much a single server can take ~</figcaption></figure><h3>2. Improving QPS: Codis-Group</h3><p>Typically, a normal application will have significantly more read operations than writes. To improve the QPS of these, we can perform <strong>Master-Slave Replication </strong>by create one or more Slaves that helps the master handle <strong>Read-only</strong> operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1b3XDZtrszWdAXVkQaU3Q.png" /><figcaption>give your Redis a BFF, perform master-slave replication</figcaption></figure><p>To maintain consistency with the Master, the master Redis will continuously push changes to the data to the Slave to be updated. This mechanism however, <strong>does not guarantee strong consistency</strong> between the Master and Slave. This was intentionally done to maintain the lowest possible latency for every operation, which is the original design consideration of Redis.</p><p>A network of Redis Servers (one Master, many Slaves) make up a single <strong>Codis Group</strong>.</p><h3>3. Improving QPS &amp; Storage capacity: Multiple Codis Groups</h3><p>We are also able to scale this out by having multiple Codis Groups, with each group storing and operating on its own unique set of keys and values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2KBWHfWkHWIFYn8KLURz6w.png" /><figcaption>Teamwork makes the dream work!</figcaption></figure><h3>Coordination — Codis Proxy</h3><p>Having multiple Codis groups requires some sort of coordinator or orchestrator to allocate which group manages which keys. That is the job of the Codis Proxy. This is done through a process called <strong>Slotting</strong>.</p><h4>Slotting</h4><p>The Slotting Algorithm maps any key to one of 1024 slots (or buckets) the following way:</p><pre>slot = CRC16(key) % 1024</pre><p>Let’s say a proxy calculates the slot of a particular key to be slot # 5</p><p>The Codis proxy then finds out which Codis group is managing keys for slot 5 (A Codis group can manage multiple slots)</p><p>After that, it forwards the query to the Codis group where it will perform the necessary operations on the group.</p><h3>So, what caused the bug?</h3><p>Let’s say you have <strong>key1</strong> which is stored in C<strong>odis group 1</strong> and <strong>key19</strong> stored in <strong>group 2</strong>.</p><p>When you execute the following redis script:</p><pre>GET_UNREAD_CONV_COUNT_LUA = `<br>   local t = {}<br>   for i, key in ipairs(KEYS) do<br>      table.insert(t, redis.call(&#39;ZCARD&#39;, key))<br>   end<br>   return t<br>`</pre><p>The script will be diverted to the group which holds the first key. I.e. if keys = {<strong>key1</strong>, <strong>key19</strong>}, script will be passed to C<strong>odis group 1</strong>, vice versa.</p><p>Codis group 1 executes the script. It manages to find <strong>key</strong> no problem. But when it tries to get the value of <strong>key19,</strong> it returns <em>nil</em> as <strong>key19</strong> is stored in <strong>Codis group 2</strong> and not <strong>group 1</strong>. Therefore it returns the wrong value of <em>NIL</em>.</p><pre>127.0.0.1:19000&gt; eval &quot;local t = {}; for i, v in ipairs(KEYS) do table.insert(t, redis.call(&#39;zcard&#39;, v)) end return t&quot; 2 key1 key19</pre><pre>1) (integer) 2<br>2) (integer) 0</pre><p>You may verify this by switching the order of the keys to see the difference!</p><pre>127.0.0.1:19000&gt; eval &quot;local t = {}; for i, v in ipairs(KEYS) do table.insert(t, redis.call(&#39;zcard&#39;, v)) end return t&quot; 2 key19 key1</pre><pre>1) (integer) 1<br>2) (integer) 0</pre><h3>Results from Redis Cluster</h3><p>When we try to run the script on a Redis Cluster, it will return this error:</p><pre>ERR CROSSSLOT Keys in request don&#39;t hash to the same slot</pre><p>At least it returns an error rather than Codis which provides a wrong result!</p><h3>How to Fix the Issue?</h3><h4>1. Use Hash Tags</h4><p>For both Codis and Redis Cluster, if there is a set of <strong><em>{}</em></strong> in the key (i.e. <strong>user_id{1234}</strong>), the proxy will calculate the slot based on the hash of the string within the bracket.</p><p>e.g. slot of <strong>user_id{1234}</strong>, <strong>conversations{1234}</strong> &amp; <strong>1234 </strong>will always map to the same slot (see <a href="https://redis.io/topics/cluster-spec">https://redis.io/topics/cluster-spec</a>)</p><p>Using Hash Tags you can ensure the keys are always mapped to the same slot and hence, same machine</p><pre>127.0.0.1:19000&gt; eval &quot;local t = {}; for i, v in ipairs(KEYS) do table.insert(t, redis.call(&#39;zcard&#39;, v)) end return t&quot; 2 key19{some_str} key1{some_str}</pre><pre>1) (integer) 1<br>2) (integer) 2</pre><h4>2. Batch the queries yourself</h4><p>This method should be used if you are already using hash tags and need to bulk access keys across slots. This is less than ideal since it delves into the distributed mechanism of Codis which should be kept separate from application logic.</p><p>What you do is first find the mapping of slots to group (i.e. slot 1 → group 1, slot 2 → group 1, slot 3 → group 2 …). In codis there is a command <strong>slotsmapping, </strong>for Redis Cluster — <strong>Cluster Slots.</strong></p><pre>127.0.0.1:19000&gt; slotsmapping</pre><pre>1) 1) 0<br>   2) 127.0.0.1:6379<br>   3)<br>   4) (nil)</pre><pre>2) 1) 1<br>   2) 127.0.0.1:6379<br>   3)<br>   4) (nil)</pre><pre>...</pre><p>Next, for every key, find the slot which it maps to. You can either do this by calculating the hash yourself, or asking Codis proxy for it via <strong>slotshashkey </strong>, <strong>Cluster Keyslot</strong> for cache cloud.</p><pre>127.0.0.1:19000&gt; slotshashkey key1</pre><pre>1) (integer) 80</pre><p>From these two pieces of data, batch keys based on the group/cluster it is destined to and send them off with the script for each batch.</p><p>Afterwards, you will still need to reassemble it in the correct order which came in the request, which is a total headache if it wasn’t already!</p><h3>Conclusion</h3><p>Did any of you, having no prior knowledge of this bug, manage to figure it out yourself?</p><p>We are so used to thinking linearly and in a single dimension, either because we are biologically engineered that way or we’ve been trained that way, that when it comes to handling multiple data we fumble easily. Perhaps that is why solving distributed/concurrency bugs are the most difficult kinds to solve!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2b7a33bf9e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go2 Sneak Preview: Generics]]></title>
            <link>https://ngtzeyang94.medium.com/go2-sneak-preview-generics-60ee3b4a730?source=rss-3d1267537130------2</link>
            <guid isPermaLink="false">https://medium.com/p/60ee3b4a730</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Ng Tze Yang]]></dc:creator>
            <pubDate>Fri, 03 Jul 2020 11:14:25 GMT</pubDate>
            <atom:updated>2020-07-21T08:25:51.794Z</atom:updated>
            <content:encoded><![CDATA[<h3>TLDR;</h3><p>Golang genetics is being planned for Go2. Here’s a sneak preview!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*RlkqwlBJS6BVA31rtBs_yA.jpeg" /><figcaption>Here We Go!</figcaption></figure><h3>Background</h3><p>A generic method is a function which objects of various types may be input as its argument. Below is an example of a Java generic method:</p><pre><strong>public</strong> <strong>static</strong> &lt; E &gt; <strong>void</strong> printArray( E[] inputArray ) {<br>   // Display array elements<br><strong>   for</strong>(E element : inputArray) {<br>      System.out.printf(&quot;%s &quot;, element);<br>   }<br>}</pre><p>Above, the method accepts arrays of any type, be it int, string, bool etc etc…</p><p>Currently Go does not allow for generics. However, earlier this month the developers of go have released an <a href="https://blog.golang.org/generics-next-step">updated design draft </a>and an <a href="https://go2goplay.golang.org/">experimentation tool </a>on playground.</p><p>Below, I’ve written a couple of examples in the playground adapted from this <a href="https://twitter.com/jboursiquot/status/1273014231873589250">twitter post</a>, that showcases the new functionality in store when generics finally got released!</p><h3>Examples</h3><h4>1. Simple generic example</h4><p>Like the Java example above, the Print method below takes a slice of generic type and prints it out.</p><pre>type User struct{<br>   UserID <strong>int<br>   </strong>Email <strong>string</strong><br>}</pre><pre>func Print(type T)(s []T) {<br>   <strong>for</strong> _, v := <strong>range</strong> s {<br>      fmt.Printf(&quot;%T, %v \n&quot;,v,v)<br>}</pre><pre>func main() {<br>   Print([]<strong>string</strong>{&quot;Hello, &quot;, &quot;playground\n&quot;})<br>   Print([]<strong>int</strong>{1,2,3})<br>   Print([]<strong>bool</strong>{true, false, true})<br>   Print([]<strong>User</strong>{<br>      {<strong>UserID</strong> : 1, <strong>Email</strong>: &quot;alice@shopee.com&quot;},<br>      {<strong>UserID</strong> : 2, <strong>Email</strong>: &quot;bob@shopee.com&quot;},<br>   })<br>}</pre><p><a href="https://go2goplay.golang.org/p/gplnV1mtkrs">https://go2goplay.golang.org/p/gplnV1mtkrs</a></p><ul><li>Take note of the new function signature of Print. T is called the<strong> type parameter</strong>.</li></ul><h4>2. A more useful generic example</h4><p>Maybe the print example was a little trivial. Below is an example generic Filter method that filters a slice based on the input filter function.</p><pre>func filter(type T)(s []T, f func(T) bool)[]T {<br>   var res []T<br>   <strong>for</strong> _, v := range s {<br>      <strong>if</strong> f(v){<br>         res = append(res,v)<br>      }<br>   }<br>   <strong>return</strong> res<br>}</pre><pre>func main() {<br>   isOdd := func(n <strong>int</strong>) bool{<br>      <strong>return</strong> n%2 == 1<br>   }</pre><pre>   isEven := func(n <strong>int</strong>) bool{<br>      <strong>return</strong> n%2 == 0<br>   }</pre><pre>   nums := []<strong>int</strong>{1,2,3,4,5,6}<br>   fmt.Println(filter(nums,isOdd))<br>   fmt.Println(filter(nums,isEven))<br>}</pre><p><a href="https://go2goplay.golang.org/p/00PeoXo3xhs">https://go2goplay.golang.org/p/00PeoXo3xhs</a></p><p>This example illustrates the best argument for using generics — abstraction of type away from implementation. For every permutation of (type, filtering algorithm), you may call the filter method and it will work as expected. Think <strong>isPalindrome</strong> for a <strong>string</strong>, <strong>isPositive</strong> for any <strong>int</strong> or any customised filter for any private struct you may instantiate!</p><h4>3. Type Constraints</h4><p>If you were asking yourself in the previous example, so what happens if you input a <strong>[]string</strong> and <strong>isOdd</strong> into the filter method? It will cause a panic! To prevent this, we can place restrictions on the input type as follow:</p><pre>type number <strong>interface</strong>{<br>   type <strong>int</strong>, <strong>int8</strong>, <strong>int16</strong>, <strong>int32</strong>, <strong>int64</strong><br>}</pre><pre>func getEven(type T number)(s []T)[]T {<br>   var res []T<br>   <strong>for</strong> _, v := range s {<br>      <strong>if</strong> v%2 == 0{<br>         res = append(res, v)<br>      }<br>   }<br>   <strong>return</strong> res<br>}</pre><pre>func main() {<br>   nums := []<strong>int</strong>{1,2,3,4,5,6}<br>   fmt.Println(getEven(nums))<br>}</pre><p><a href="https://go2goplay.golang.org/p/9mP9Dx4R-Is">https://go2goplay.golang.org/p/9mP9Dx4R-Is</a></p><p>By restricting the type argument to <strong>ints, </strong>we allow the compiler to catch wrongly typed calls to our <strong>getEven</strong> Function, preventing a runtime error.</p><h4>4. What about custom types?</h4><p>Let’s say you have a custom type such as <strong>type customInt int</strong>, will it fulfil the type constraints?</p><pre>type number <strong>interface</strong>{<br>   type <strong>int</strong>, <strong>int8</strong>, <strong>int16</strong>, <strong>int32</strong>, <strong>int64</strong><br>}</pre><pre>type customInt int64</pre><pre>func getEven(type T number)(s []T)[]T {<br>   var res []T<br>   <strong>for</strong> _, v := range s {<br>      <strong>if</strong> v%2 == 0{<br>         res = append(res, v)<br>      }<br>   }<br>   <strong>return</strong> res<br>}</pre><pre>func main() {<br>   nums := []customInt{1,2,3,4,5,6}<br>   fmt.Println(getEven(nums))<br>}</pre><p><a href="https://go2goplay.golang.org/p/spSHMPEcv4E">https://go2goplay.golang.org/p/spSHMPEcv4E</a></p><p>The answer is <strong>YES!</strong></p><h4>5. Interface Constraints i.e. Stringer</h4><p>Not only can you out constraints on the type, you may also constraint it by interfaces:</p><pre>type User struct{<br>   UserID <strong>int<br>   </strong>Email string<br>}</pre><pre>func (u User) String() string {<br>   <strong>return</strong> strconv.Itoa(u.UserID)+&quot;:&quot;+u.Email<br>}</pre><pre>func stringify(type T fmt.Stringer)(v T)string {<br>   <strong>return</strong> v.String()<br>}</pre><pre>func main() {<br>   user := User{<br>      <strong>UserID</strong>: 1,<br>      <strong>Email</strong>: &quot;alice@shopee.com&quot;,<br>   }<br>   fmt.Println(stringify(user))<br>}</pre><p><a href="https://go2goplay.golang.org/p/5dTjhc5yQM_U">https://go2goplay.golang.org/p/5dTjhc5yQM_U</a></p><p>Recall the <strong>fmt.Stringer</strong> interface:</p><pre>type Stringer interface {<br>    String() <a href="https://golang.org/pkg/builtin/#string">string</a><br>}</pre><p>In the above example, we constraint the input for <strong>stringify</strong> to all types which implements <strong>String(). </strong>However, this is kind of similar to constraining the argument directly and I am still unsure of its benefits?</p><p><em>edit: Found the answer from an expert — Implementing the Interface Constraint on the function allows you to pass in a Slice or a Map with a Type that fulfils the constraint.</em></p><pre>1) func stringify(type T fmt.Stringer)(v []T)string<br>2) func stringify(v fmt.Stringer)string</pre><p><em>i.e. in signature (1) we only need to implement String() to be able to pass it into the function, whereas in signature (2), we also need to ensure that the slice of types also implements String() which is a little inconvenient</em></p><h4>6. Comparable</h4><p>Comparable is a predeclared type constraint. The comparable type permits the use of == and != on the type object.</p><pre>func Index(type T comparable)(s []T, x T) <strong>int</strong> {<br>   <strong>for</strong> i, v := range s {<br>      // v and x are type T, which has the comparable<br>      // constraint, so we can use == here.<br>      <strong>if</strong> v == x {<br>         <strong>return</strong> i<br>      }<br>   }<br>   <strong>return</strong> -1<br>}</pre><pre>func main() {<br>   fmt.Println(Index([]<strong>int</strong>{1,2},2))<br>}</pre><p><a href="https://go2goplay.golang.org/p/ZRraCRpfk3o">https://go2goplay.golang.org/p/ZRraCRpfk3o</a></p><h4>7. Generic types</h4><p>There are also generic types. A custom type that can be used by multiple types.</p><pre>type User struct{<br>   UserID <strong>int<br>   </strong>Email <strong>string<br></strong>}</pre><pre>type Vector(type T) []T</pre><pre>func (v *Vector(T)) push(element T) {<br>   *v = append(*v, element)<br>}</pre><pre>func main() {<br>   var intVec Vector(<strong>int</strong>)<br>   <strong>for</strong> _,e := range []<strong>int</strong>{1,2,3} {<br>      intVec.push(e)<br>   }</pre><pre>   fmt.Println(intVec)</pre><pre>   users := []User{<br>      {UserID: 1, Email: &quot;alice@shopee.com&quot;},<br>      {UserID: 2, Email: &quot;bob@shopee.com&quot;},<br>   }</pre><pre>   var userVec Vector(User)<br>   <strong>for</strong> _,e := range users {<br>      userVec.push(e)<br>   }<br>   fmt.Println(userVec)<br>}</pre><p><a href="https://go2goplay.golang.org/p/NJOaNjz_7kj">https://go2goplay.golang.org/p/NJOaNjz_7kj</a></p><p>Take note of the way we instantiate intVec</p><pre>var intVec Vector(int)</pre><p>We are creating a <strong>Vector</strong> of type <strong>int</strong>. Likewise, you can also create Vectors of <strong>bool</strong> type, <strong>string</strong> type etc…</p><h3>Final notes</h3><ul><li>Generics is still undergoing review. May still take awhile for such functionality to be live (probably in go2)</li><li>I barely skimmed through the design doc. Do add on to the page if you have other good examples!</li><li>Also feel free to point out any mistakes I may have made above! It will be much appreciated</li></ul><p>If you want to find out more check out this comprehensive repo by ArdanLabs: <a href="https://github.com/ardanlabs/gotraining/tree/3488830f299d43bc93d5110ce82233723e916dd5/topics/go/generics">https://github.com/ardanlabs/gotraining/tree/3488830f299d43bc93d5110ce82233723e916dd5/topics/go/generics</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60ee3b4a730" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>