<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[The Software Firehose - Medium]]></title>
        <description><![CDATA[Unrefined, rapid-fire snippets of practical software engineering - Medium]]></description>
        <link>https://medium.com/the-software-firehose?source=rss----bfe766a2c671---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>The Software Firehose - Medium</title>
            <link>https://medium.com/the-software-firehose?source=rss----bfe766a2c671---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:40:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/the-software-firehose" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Information Hiding, Encapsulation and Modularity of Software]]></title>
            <link>https://medium.com/the-software-firehose/information-hiding-encapsulation-and-modularity-of-software-281a16ced?source=rss----bfe766a2c671---4</link>
            <guid isPermaLink="false">https://medium.com/p/281a16ced</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[prototype]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Shaaz Ahmed]]></dc:creator>
            <pubDate>Sun, 25 Mar 2018 19:12:07 GMT</pubDate>
            <atom:updated>2019-01-19T21:36:24.414Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding the motivation and history of these notions and their relationships</h4><p><strong>UPDATE:</strong> <em>Since writing this, my understanding of these concepts have either changed or found enhanced clarity in certain cases. Two papers that have significantly influenced my thoughts in this area are</em><strong><em> Programming as Theory Building </em></strong><em>by Naur and </em><strong><em>Out of the Tar Pit </em></strong><em>by Moseley and Marks. I’m certain that it would be a better use of your time to skip my blog post, and read those two papers instead. I’m not removing this (rather incoherent) post.</em></p><p>I’ve been interested in the role of information hiding in producing good code, and the goals that information hiding tries to achieve. I’m curious about other ways in which the same goal can be achieved. We will talk about the origins of the concept of information hiding, the motivation behind it, concepts similar to it, and briefly touch upon how functional programming strives to achieve the similar goals.</p><p>Like most of the little academic knowledge of software that I’ve acquired, I first came across the concept of information hiding in <a href="https://mitpress.mit.edu/sicp/full-text/sicp/book/node54.html">the section called “<em>The Costs of Introducing Assignment</em>”</a> of the classic book <em>Structure of Interpretation of Computer Programs</em>. Information hiding, encapsulation and modularity are related concepts that are often loosely tossed around together while talking about good design principles, so let’s first examine what these words mean to the best of our ability.</p><h4>Definitions of Information Hiding, Encapsulation &amp; Modularity</h4><p><strong>Information Hiding:</strong></p><p>Let’s begin by taking a look at the <a href="https://en.wikipedia.org/wiki/Information_hiding">Wikipedia definition</a> for information hiding:</p><blockquote>[…] information hiding is the principle of segregation of the design decisions in a computer program that are most likely to change, thus protecting other parts of the program from extensive modification if the design decision is changed. […]</blockquote><p>That sentence describes what information hiding entails, including one concrete benefit of employing information hiding.</p><p><strong>Encapsulation:</strong></p><p>In his book <em>Object-Oriented Analysis and Design with Applications</em>, Grady Booch defines encapsulation as:</p><blockquote>the process of compartmentalizing the elements of an abstraction that constitute its structure and behavior; encapsulation serves to separate the contractual interface of an abstraction and its implementation.</blockquote><p>The term encapsulation and information hiding are sometimes used interchangeably — however, there isn’t any unanimous agreement that they are identical. Many people try to reconcile these differences by claiming that information hiding is the principle and that encapsulation is the technique. Frankly, I was under the impression that the reverse was true — that encapsulation was the principle and information hiding was the technique. Well, it doesn’t look like we’ll get far with that chain of reasoning — it is probably more useful to look at the consequences of these two different ideas than to understand what they are.</p><p><strong>Modularity</strong></p><p>I believe that the goals of modularity encompasses the goals behind both encapsulation and information hiding, but seeking a definition leads us to even more nebulous notions.</p><p>Wikipedia defines <a href="https://en.wikipedia.org/wiki/Modularity">modularity</a> as<em> ‘the degree to which a system’s components may be separated and recombined’. </em>Other definitions may omit the ‘recombined’ aspect. However, Wikipedia also has other (but similar) definitions for modularity. It describes <a href="https://en.wikipedia.org/wiki/Modular_programming">modular programmin</a>g as ‘<em>a software design technique that emphasizes separating the functionality of a program into independent, interchangeable modules, such that each contains everything necessary to execute only one aspect of the desired functionality’. </em>In yet another place, it defines <a href="https://en.wikipedia.org/wiki/Software_design">modularity in the context of software design</a> as ‘<em>a logical partitioning of the “software design” that allows complex software to be manageable for the purpose of implementation and maintenance. The logic of partitioning may be based on related functions, implementation considerations, data links, or other criteria’.</em></p><p>Personally, I find that a much easier way to understand modularity is by contrasting it with what seems to be its logical antonym — interdependence, or strong coupling. These distinctions between modularity and interdependence are nicely illustrated and compared on <a href="https://www.christenseninstitute.org/interdependence-modularity/">the Christensen Institute website</a>.</p><h4>Effects of Information Hiding, Encapsulation &amp; Modularity</h4><p>As Nassim Nicholas Taleb talks about in his book <em>Antifragile</em>, it is often easier to understand the effects of a principle than the principle itself.</p><p>To understand the effects of information hiding, one need not look further than its history.</p><h4><strong>History of Information Hiding</strong></h4><p>Information hiding was first introduced in David Parnas’ 1972 paper: “<a href="http://repository.cmu.edu/cgi/viewcontent.cgi?article=2979&amp;context=compsci">On the criteria to be used in decomposing systems into modules</a>”. In that paper, he suggests splitting programs into different modules, and listing down and hiding within a module the<em> ‘design decisions most likely to change’</em>, so that you need to change just one module when one of those decisions change. On a tangential note, Parnas wrote <a href="http://extras.springer.com/2002/978-3-642-59413-7/3/rom/pdf/Parnas_new.pdf">a paper with the same name</a> in 2002, clarifying aspects of his older paper and what he has learnt since then, including a few interesting comments on OOP and ADTs.</p><p>In the 1972 paper, Parnas cites his goal as modularity (as evident from the title of the paper). But what are the benefits of modularity?</p><h4><strong>The Upsides of Modularity</strong></h4><p>In his book <em>Working Effectively with Legacy Code</em>, Michael Feathers cites four reasons to change code: adding a feature, fixing a bug, improving the design and optimizing resource usage. The goal of modularity is to ease the process of changing code.</p><ol><li>Modularity <strong>allows you to swap out one implementation of a part with another</strong>, easing optimization of resource usage and also replacing parts of the code with minimal changes. This is what Parnas tried to achieve with information hiding— hide design decisions most likely to change within a module.</li><li>Modularity <strong>lets other humans easily identify which parts to change and where bugs may lurk</strong>, because it establishes a conceptual barrier around a collection of things. In this respect, modularity is the practical realization of encapsulation, i.e. modules are mappings of encapsulated abstractions to real systems. If I know that I’ve strongly encapsulated my code, then I have to worry much lesser about whether change in one module mandates change in another.</li><li>As an extension of point #2, modularity tries to <strong>work around the inability of human beings to keep many things in their head</strong>. In one of the most cited papers in psychology, “<a href="https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two"><em>The Magic Number Plus or Minus Seven: Some Limits on our Capacity for Processing Information</em></a>”, cognitive psychologist George Miller argues that the number of objects an average human can hold in working memory is 7 ± 2. This principle necessitates the modularity of a system for the sake of understanding the system easily, and consequently, ease of maintenance. Martin Fowler expresses this in his comment: “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”</li><li>Modularity done right can lead to <strong>high reusability</strong> of the components of your code. In this context, the benefits of modularity are multiplicative — if you’re able to easily reuse a module that took you two weeks to implement with an hour’s effort, you’ve practically scaled your productivity non-linearly by saving time for both you and the people who you’re writing software for.</li></ol><p>However, for the last point to be valid, your modules should possess certain positive traits. In Bartosz Milewski’s online series ‘<a href="https://bartoszmilewski.com/2014/11/04/category-the-essence-of-composition/"><em>Category Theory for Programmers</em></a>’, he posits that modules are useful when the information required to compose them grows slower than the information required to implement them. “<em>The idea is that, once a chunk is implemented, we can forget about the details of its implementation and concentrate on how it interacts with other chunks.</em>” This is also what good encapsulation can help you achieve.</p><p>I’m sure I’ve missed out quite a few upsides (and potential downsides) of modularity. However, I hope the discussion has helped better understand the intertwined relationships between modularity, encapsulation and information hiding.</p><h4>Functional Programming</h4><p>Functional programming tries to achieve the goal of modularity in different ways. A concrete example of the reusability via modularity in functional programming is provided in the chapter of <em>Structure and Interpretation of Computer Programs</em> called<a href="https://mitpress.mit.edu/sicp/full-text/sicp/book/node74.html"> <em>“Modularity of Functional Programs and Modularity of Objects”</em></a><em>. </em>If I were to read only two things on modularity, it would be that chapter and the previous one mentioned in the introduction to this post.</p><p>However, that chapter does illustrate that functional programming does not magically take away the issues software raises.</p><p>Another seminal paper that outlines the differences in approach of functional programming is <a href="https://github.com/papers-we-love/papers-we-love/blob/master/design/out-of-the-tar-pit.pdf">Out of the Tarpit</a>. It illustrates how reasoning about state is much harder than reasoning about pure transformations, and consequently reducing ease of changing code and identifying bugs. It also raises some interesting points, such as the fact that long-living code is often written once and read many times (to simply grossly). This means that optimizing for readability may have higher pay offs in the future than optimizing for immediate ‘write-ability’ of the code.</p><ul><li><a href="https://medium.com/the-software-firehose">The Software Firehose</a></li><li><a href="https://medium.com/hacking-learning">Hacking Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=281a16ced" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-software-firehose/information-hiding-encapsulation-and-modularity-of-software-281a16ced">Information Hiding, Encapsulation and Modularity of Software</a> was originally published in <a href="https://medium.com/the-software-firehose">The Software Firehose</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Canceling requests in Go using context]]></title>
            <link>https://medium.com/the-software-firehose/canceling-requests-in-go-using-context-b4b28a118da8?source=rss----bfe766a2c671---4</link>
            <guid isPermaLink="false">https://medium.com/p/b4b28a118da8</guid>
            <category><![CDATA[cancel-request]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[cancel-goroutine]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[practical-tips]]></category>
            <dc:creator><![CDATA[Shaaz Ahmed]]></dc:creator>
            <pubDate>Fri, 23 Mar 2018 19:47:10 GMT</pubDate>
            <atom:updated>2019-01-19T21:35:45.857Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, we wrote a small client library that simplifies making parallel HTTP requests in Golang. One of the required behaviours was the cancellation of all requests that took more time than the specified collective request timeout. Here’s what the library’s API looks like:</p><pre>timeout := 200 * time.Millisecond<br>bulkClient := NewBulkHTTPClient(timeout)</pre><pre>requestOne := http.NewRequest(http.MethodGet, &#39;&#39;)<br>requestTwo := http.NewRequest(http.MethodGet, &#39;&#39;)</pre><pre>bulkRequest := NewBulkHTTPRequest()<br>bulkRequest.Add(requestOne)<br>bulkRequest.Add(requestTwo) // requestTwo is *http.Response</pre><pre>responses, errors := bulkClient.Do(bulkRequest)<br>// return signature is []*http.Response, []errors</pre><p>To cancel the HTTP requests, we initially explored a couple of options for the cancellation signal:</p><ul><li>using a time.After() channel within our client library and listening to it. When we receive a timeout signal, we stop listening to the response channel by exiting a select block.</li><li>using context.WithTimeout() , attaching that to the request with request.WithContext() and then listening to thectx.Done() . When we receive a signal on the channel, we stop listening to the response channel.</li></ul><p>We chose to go with context.WithTimeout() approach since it is also used by net/http ‘s http.Client.Do() method to terminate the request at different stages of the client connection process. We weren’t very impressed with some of the consequences of using context.Context , but decided to stick with it for lack of a better and convenient alternative. There’s a good deal written about the downsides of context (<a href="https://faiface.github.io/post/context-should-go-away-go2/">here</a> and <a href="https://dave.cheney.net/2017/08/20/context-isnt-for-cancellation">here</a>).</p><h4>Usage, and the scope of a request</h4><p>Before we discuss some of the gotchas and subtleties around the usage context to cancel HTTP requests, let’s take a practical example of how we can use it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63f8cace18c91b89e4c81aca06f04400/href">https://medium.com/media/63f8cace18c91b89e4c81aca06f04400/href</a></iframe><p>Note that I’ve mentioned the ioutil.ReadAll() step above —<strong> that’s because the context is actually relevant until </strong><strong>response.Body has been read</strong>. If the context timeout occurs between lines 11 and 14, readErr will be non-nil and will contain a context canceled error along with a correctly read responseByteArray. I couldn’t find any documentation regarding this, except for <a href="https://groups.google.com/forum/#!topic/golang-nuts/2FKwG6oEvos">this thread on the golang-nuts Google group</a> which mentions that the scope of a request ends only when the response is read. This makes sense for streamed responses, but it escapes me how it’s a good idea for normal ones.</p><h4>Stages of HTTP Requests</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8nT1H5-GngbLdvh8651K4g.png" /><figcaption>Different stages of request cancellation using timeouts(from <a href="https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/">this article</a>)</figcaption></figure><p>Request cancellation isn’t possible for in-flight requests. I’d recommend reading <a href="https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/">this article</a> for understanding the different stages of an HTTP connection with respect to net/http and at what stages cancellations can occur. We’ve only observed requests being cancelled before and during connection establishment, but not after a successful connection establishment. However, we didn’t explore this further as this was sufficient for our cases.</p><h4>Deprecated Methods</h4><p>There were a couple of other methods we came across for canceling requests, which we learnt were deprecated. This is briefly explained in the comments of net/http/client.go on <a href="https://github.com/golang/go/blob/7662e6588c9433f1219e0a0c46bc563ba3f93f98/src/net/http/client.go#L282">lines 282–285</a>.</p><pre>// As background, there are three ways to cancel a request:<br>// First was Transport.CancelRequest. (deprecated)<br>// Second was Request.Cancel (this mechanism).<br>// Third was Request.Context.</pre><ul><li><a href="https://medium.com/the-software-firehose">The Software Firehose</a></li><li><a href="https://medium.com/hacking-learning">Hacking Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4b28a118da8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-software-firehose/canceling-requests-in-go-using-context-b4b28a118da8">Canceling requests in Go using context</a> was originally published in <a href="https://medium.com/the-software-firehose">The Software Firehose</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seeing Responsibilities in Object-Oriented Code]]></title>
            <link>https://medium.com/the-software-firehose/seeing-responsibilities-in-object-oriented-code-eca377a7e7e?source=rss----bfe766a2c671---4</link>
            <guid isPermaLink="false">https://medium.com/p/eca377a7e7e</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[object-oriented]]></category>
            <category><![CDATA[code-quality]]></category>
            <category><![CDATA[practical-tips]]></category>
            <dc:creator><![CDATA[Shaaz Ahmed]]></dc:creator>
            <pubDate>Thu, 22 Mar 2018 19:54:39 GMT</pubDate>
            <atom:updated>2019-01-19T21:34:58.988Z</atom:updated>
            <content:encoded><![CDATA[<h4>Some thoughts on splitting the 1000-line class</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*10AZbTavAgwbOVGE06Xx7A.jpeg" /></figure><h4>Motivation</h4><p>In the past few months, I’ve had to work with three different classes longer than a 1000 lines of code in Java and Ruby on Rails codebases. Although being a <em>1000-line class</em> by itself is a superficial concern and not necessarily problematic, the consequences that often accompany it may be a latent productivity leech for programmers maintaining that code.</p><p>Exemplar negative consequences include code that is difficult to comprehend or reason about, often filled with intertwined concerns that make it harder to spot mistakes and redundancy in the code. Large classes also obscure the flow of the operations being executed and make it harder to decipher where a particular behaviour should be added. In the object-oriented paradigm where mutability is more a norm than a sin, this can also lead to difficulty in unit testing behaviours as it becomes more involved to replicate the state of the object in question to pull it into a test harness.</p><h4>‘Rigidity’ of Object Models</h4><p>In contrast with dynamically-typed functional programming languages where behaviours or functions can exist largely independently of the types or objects, the object-oriented programming languages require that your function is <em>strongly</em> associated to an entity. I believe these are two polar approaches to dealing with change — the former deals with it by embracing it, while the latter deals with it by rejecting it.</p><p>Functional programming deals with the problem of change by being very malleable and having fewer implicit constraints around where a piece of code belongs, while object-oriented programming deals with by leaning in the other direction — by being rigid and being less permissive of moving around code. However, leaning too far towards either direction can be damaging, and a programmer needs to exercise discipline to preserve structure in a codebase that doesn’t impose the coupling of behaviours and domain entities and to preserve flexibility in a codebase that imposes it. To go further down this train of thought without the specific comparisons of these paradigms, I would recommend you watch <a href="https://www.youtube.com/watch?v=x9pxbnFC4aQ">Zach Tellman’s fantastic talk “On Abstraction”</a>.</p><p>However, in the rest of this post, I will be covering how to tackle the latter problem — identifying large classes and refactoring them to manageable sizes to improve the malleability and reusability of your code.</p><p>Of course, I’m in no way claiming that it’s impossible to move methods around in an object-oriented programming language. However, I admit that it is often more difficult to do so in such languages due to the overhead of creating an extra class to move it or due to the complications that mutable state brings with it.</p><h4>‘Does this method belong here?’</h4><p>While adding a certain method to a class, I often find myself asking the question “does this method belong here?”. In the case of a really large class, this question by itself is <strong><em>not</em></strong><em> </em>sufficient. This innocuous litmus test overlooks the subjective nature of object models and the different levels of granularity possible while modelling domain objects. The fact that many correct models exist for the same problem arise from the fact that a higher level of abstraction can exist within object models to accommodate entities at a lower level of abstraction. Although many people are devout believers of the <em>Single Responsibility Principle</em>, they often slip up on the fact that the idea of a responsibility can be nebulous.</p><p>To refrain from talking entirely in the abstract, I’ll describe an example. Take for example, the order management system of a airline ticketing company. Initially, the company’s website takes bookings only via its website and app, and you decide to call a class Booking (one could argue that this is a bad choice). Later, the company expands into hotel bookings, which involves different business logic, and you ask yourself “does this method belong here?”. You answer, “yes, it’s related to the booking”. And as the company grows into other domains, your Booking class begins to do more than it was ever meant to, dealing with a wide variety of domains.</p><p>With the viewpoint that objects should model life-cycles and not real-word entities, one could have chosen a different class name. Also, another dimension in which the class can grow is the number of state transitions it undergoes. It is possible, and sometimes more manageable to create a separate class for a ConfirmedBooking or a CancelledBooking if there is a large amount of business logic associated with these state transitions. One could also drop in interfaces into this picture to deal with these different dimensions of change and growth, and increase the adaptability of the code.</p><p>However, I’m going to assume you’re already stuck with a 1000-line class (whether you knew it all along or whether it just snuck up on you) and describe a few approaches to tease apart concerns in a class by ‘seeing responsibilities’ where responsibilities are entangled and spread across more lines than your brain can hold on to.</p><h4>Approaches</h4><p>The approaches listed here are mentioned in Michael C. Feathers’ book <em>Working Effectively with Legacy Code</em>. Steve from Nilenso recommended that I read this book, and I’ve found the lessons from it extremely valuable. Before coming to the conclusion that you don’t really work with legacy code, do consider the fact that code that you yourself write can turn out to be legacy code when you come back to modify it a year later.</p><p>I recommend reading the relevant chapter from the book for more examples and a better understanding — it’s quite a lucid read. I’ll quickly go through the methods described in Feathers’ book here.</p><ul><li>Group methods</li><li>Look at hidden methods</li><li>Identify decisions that can change</li><li>Identify and group internal relationships</li><li>Describe the main purpose</li></ul><h4>1. Group Methods</h4><p>This was one of the most effective methods, for my use case. This method involves listing the different methods in your large class, and trying to group them in different ways. If you see different even-sized groups (or a mix of even-sized and tiny atomic units) emerging, you can try to identify different responsibilities and refactor your code for better separation.</p><h4>2. Look at hidden methods</h4><p>The private functions in your code can hide a lot of distinction in responsibilities, and it’s probably worth looking at them to see if you can tease out a large chunk of separate behaviours. I haven’t found this particularly useful yet.</p><h4>3. Identify decisions that can change</h4><p>It’s possible to overdo this method that lead premature abstractions, so tread with caution. Basically, look at your code, identify critical decisions around domain entities that have the possibility to change. The example of the ticketing company discussed above is a relevant one — you could sometimes foresee such changes.</p><h4>4. Identify and group internal relationships</h4><p>This method also proved to be very helpful for my use case. You look for how your instance methods and instance variables interact, and identify which groups of methods interact with which groups of instance variables. If you find that some instance variables are not at all used by some methods but are used a by another group of methods, then it’s likely that there’s a responsibility in there waiting to be teased out.</p><h4>5. Describe the main purpose of the class</h4><p>Although describing the main purpose of the class in a single sentence as a solution to nebulous responsibilities may seem like a cyclical argument, it may turn out to be effective if you didn’t get it right the first time around when you created the class or if you have developed a new perspective on your domain since you first created the class. Ensure that there are no conjunctions in the sentence that you use to describe the purpose, and that your implementation involves nothing more or nothing less than the described purpose.</p><ul><li><a href="https://medium.com/the-software-firehose">The Software Firehose</a></li><li><a href="https://medium.com/hacking-learning">Hacking Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eca377a7e7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-software-firehose/seeing-responsibilities-in-object-oriented-code-eca377a7e7e">Seeing Responsibilities in Object-Oriented Code</a> was originally published in <a href="https://medium.com/the-software-firehose">The Software Firehose</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recursion, Tail-call Optimization & Currying]]></title>
            <link>https://medium.com/the-software-firehose/methods-of-functional-programming-using-python-or-javascript-b3309bb2c807?source=rss----bfe766a2c671---4</link>
            <guid isPermaLink="false">https://medium.com/p/b3309bb2c807</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[prototype]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[common-lisp]]></category>
            <dc:creator><![CDATA[Shaaz Ahmed]]></dc:creator>
            <pubDate>Thu, 22 Mar 2018 19:53:33 GMT</pubDate>
            <atom:updated>2019-01-19T21:37:44.189Z</atom:updated>
            <content:encoded><![CDATA[<h4>A hodgepodge of 3 functional programming concepts</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*s6IuyLoehtSqDApWUPmCLg.png" /></figure><p><em>(Edit: I initially began writing this post with the intention of compiling an exhaustive list of ‘methods of functional programming’ known to me. Since then, I’ve done some Real</em>™<em> functional programming, and it’s become ever more evident to me that functional programming is a dense paradigm and its ‘methods’ are merely by-products of the core values it promotes. Nevertheless, this is probably a good introductory read for a beginner familiar with procedural programming.)</em></p><p>There are many methods derived from functional programming that could really add to the common programmer’s repertoire of tricks. These methods don’t reach a wide audience because they are often found in academic literature or among communities of less common languages like Lisp, Haskell etc. Here I explain a few of them using two popular languages: Python and JavaScript. We won’t discuss many of the technical aspects, but we’ll see how they come in handy.</p><p><em>In writing this article, it has become clear to me that there has to be a separate “Why functional programming?” article. I won’t be writing that. If you’re curious, Google’s your friend. I’d recommend the paper </em><a href="https://github.com/papers-we-love/papers-we-love/blob/master/design/out-of-the-tar-pit.pdf"><em>Out of the Tarpit</em></a><em> — one of the classics.</em></p><h4>A few notes before we start</h4><p>We’re about to discuss ‘programming paradigms’, which is not ‘code’ — i.e. they’re not concrete implementations of a concept in a specific programming language. These methods are independent of any programming language. However, this means that some of the methods cannot be used in some languages due to the implementation details of that language.</p><p>Also, many of these methods are not exclusive to functional programming — infact, you might find that you use many of the methods regularly even though you’re not familiar with functional programming. However, there maybe methods which are necessary conditions for programs to be considered functional.</p><h4>Recursion vs Iteration</h4><p>You’re probably thinking: “Hey, that’s not a functional programming concept. This is basic stuff.”</p><p>It definitely is. But in the case of functional programming, you have to use recursion not because <em>you can</em>, but you because <em>you have to</em>. (At least, in the strictest definition of functional programming. It’s not technically <em>disallowed, </em>but it’s discouraged.)</p><p>As explained in another article, in functional programming we care about purity — i.e. lack of side-effects. What this means is, you try to structure your code as a function that takes in an input, and gives out an input. We don’t change state (put another way, we don’t declare global variables, <em>wherever possible</em>). There are many reasons for doing so, such as referential transparency, better parallelizability, avoiding non-obvious dependencies etc. (read about the reasons here).</p><p>You’re probably thinking: “But that’s impossible! How can you avoid declaring globally-scoped variables?”</p><p>Well, it seems there are tons of methods to do so. For example, recursion.</p><p>If you can’t declare a variable, then how do you implement a loop? Using a loop counter involves state — so how do you work around that? Recursion, of course.</p><p>Let’s take an example in Javascript [<em>since the different is not as clear in Python</em>]:</p><pre>var n = 10;<br>var ans = 1;</pre><pre>for (i = 1; i &lt;= n; i++) {<br>    ans = i * ans;<br>}</pre><pre>console.log(ans);</pre><p>Now, this is a simple bit of code that finds the value of n! (n-factorial, i.e. 3! = 3 x 2 x 1). How would you do this recursively, without declaring the counter variable i?</p><pre>function factorial(n) {<br>    if (n != 0) {<br>        return n * factorial(n-1);<br>    }<br>    <br>    else {<br>        return 1;<br>    }<br>}</pre><pre>factorial(10);</pre><p>Clearly, we achieved the same result without declaring variables. Now, a word of caution— wait, we’ll take that up in the next section.</p><h4>Tail-call Optimization</h4><p>I deliberately picked the factorial example above to demonstrate tail-call optimization. When you use the recursive solution to find the factorial of a really large number n, you may find that your system runs out of memory. This is quite obvious from the function, because the actual multiplication is deferred until all values are obtained by the return clause. For example, for 10!: 10*factorial(9) is expanded to 10*9*factorial(8) and so on — all the numbers until 1 have to be obtained before this can be multiplied.</p><p>So, it seems like the iterative solution is the better one. But there’s a better way around! And that method is called tail-call optimization.</p><p>Here’s the tail-call optimized version of the above recursive solution [<em>note that the language has to support this — JS ES2015+ does, but Python doesn’t</em>]:</p><pre>function factorial (n) {<br>    function fact(num, acc) {<br>        if (n &lt; 2) {<br>            return acc;<br>        } <br>        else {<br>            return fact(n-1, n * acc);<br>        }<br>    }<br><br>    return fact(n, 1);<br>}</pre><p>Now, this gives us the same result without the memory issues. We worked around the issue of delayed evaluation by using an accumulator. This example should convince you that there’s often a work-around even though some methods in FP may seem unreasonable or impossible in practice.</p><h4>Currying</h4><p>Now this is one of my favourite methods for its sheer practical use: if used properly, this method can reduce repetition in your code and enhance reusability, and also make your code easier to understand (<em>if used judiciously though!).</em></p><p>Currying in it’s simplest form, is simply about replacing a function f(a,b,c) that takes 3 variables (a, b and c) with a function g(a)(b)(c). Say what? Here’s an example in Python:</p><pre>&gt; def f(a,b,c):<br>      return a + b + c</pre><pre>&gt; f(1,2,3)<br>6</pre><p>can be replaced with:</p><pre>&gt; def g(a):<br>      def g1(b):<br>          def g2(c):<br>              return a + b + c<br>          return g2<br>      return g1</pre><pre>&gt; g(1)(2)(3)<br>6</pre><p>Not clear how that happened? Maybe this will help:</p><pre>&gt; g(1)<br>&lt;function g1 at 0x1007f7230&gt;</pre><pre>&gt; g(1)(2)<br>&lt;function g2 at 0x1007f7410&gt;</pre><pre>&gt; g(1)(2)(3)<br>6</pre><p>From the above code snippet, one thing should be clear: calling g(1) generates a function, and so does g(1)(2) which gives the final function g2 in the chain. When this function is called, we get the result 6.</p><p>So how is this useful, you ask?</p><p>Currying allows you to ‘customize’ functions by wrapping it in even more general functions — i.e. to get specialized functions from more generalized functions.</p><p>In languages like Python where functions are also first-class citizens, you should remember that functions can be arguments.</p><p>Let’s take an example:</p><pre>&gt; def g(function1):<br>      def g1(list):<br>          def g2(key):<br>              return a(b[c])<br>          return g2<br>      return g1</pre><pre>&gt; g(round)(listofnumbers)(3)</pre><p>Just look at how reusable that is: you’re applying the <strong><em>round</em></strong> function to the elements in a list called <strong><em>listofnumbers </em></strong>with key <strong><em>3</em></strong><em>. </em>You can change all of it — you can change the function, the list, and the key. This means you don’t have to code it again in case you want to use a different function, or a different list. If you use this with a map function, the possibilities are endless!</p><blockquote><em>[This post is </em><strong><em>incomplete</em></strong><em>. I didn’t know where I was taking it, and I still don’t. I don’t agree with all of the content I wrote a while back — but I doubt that really matters. To steal a line from Zuckerberg:</em> if you’re not embarrassed by your first product release, you’ve released it too late.<em>]</em></blockquote><ul><li><a href="https://medium.com/the-software-firehose">The Software Firehose</a></li><li><a href="https://medium.com/hacking-learning">Hacking Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3309bb2c807" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-software-firehose/methods-of-functional-programming-using-python-or-javascript-b3309bb2c807">Recursion, Tail-call Optimization &amp; Currying</a> was originally published in <a href="https://medium.com/the-software-firehose">The Software Firehose</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Text Adventure Game in CLISP]]></title>
            <link>https://medium.com/the-software-firehose/architecting-a-text-adventure-game-2e0dc4d49812?source=rss----bfe766a2c671---4</link>
            <guid isPermaLink="false">https://medium.com/p/2e0dc4d49812</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[common-lisp]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[game-development]]></category>
            <category><![CDATA[projects]]></category>
            <dc:creator><![CDATA[Shaaz Ahmed]]></dc:creator>
            <pubDate>Thu, 22 Mar 2018 19:49:41 GMT</pubDate>
            <atom:updated>2019-12-20T09:37:16.363Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, I’ll discuss a simple command-line text adventure game, called “The Wizard’s Adventure”. The game’s code base is really small (&lt; 200 lines), and these few lines of code have been split into modules of even fewer lines based on function.</p><p>By the end of this article, I hope you end up having a good idea about:</p><ul><li>The game loop, and a way to implement it</li><li>The different functional blocks one has to code while designing a game and how these blocks work together</li><li>Separating logic such as UI actions, game state and game scene describers</li></ul><p>The code is available at <a href="https://github.com/shaaza/the-wizard-game">github.com/shaaza/the-wizard-game</a>. It’s written in Common Lisp (CLISP), which you may not be familiar with. However, I feel it’s very well commented and structured, and the purpose of each block of code can be understood even by someone who doesn’t understand the language. You can also read this post without referring to the code.</p><h4><strong>The game:</strong></h4><p>While talking about creating games, I like to use the MDA approach. Mechanics give rise to Dynamics that create Aesthetics, i.e. the experience of the game. In this article, we are mostly concerned about the mechanical aspects of the game.</p><p><em>So what’s this game about?</em></p><p>In this game, the player is a wizard’s apprentice. The player explores the wizard’s house. In the complete version of the game, the player will be able to solve puzzles and win a magical donut.</p><p>Here’s a screenshot that takes us through the initial stages of gameplay:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J67MvPEo6GqQj7u3B3qRbg.png" /><figcaption>A screenshot of the gameplay</figcaption></figure><h4><strong>The game loop:</strong></h4><p>As discussed in the article on modelling worlds, most games have the same conceptual main loop[1]:</p><pre>while (game isn&#39;t over):<br>    get user input<br>    respond to user input<br>    show stuff to user</pre><p>The game loop in our case is implemented in the form of the game’s REPL, i.e. the game Read-Eval-Print Loop.</p><p>In our game, read, eval and print are separate functions called game-read, game-eval and game-print, all housed under one enclosing function called game-repl. Thus, game-repl calls game print, which takes as argument the resuts of game-eval, which in turn takes as argument game-read. We can express this like so, using Python equivalent code (hence it’s easier to understand):</p><pre>def <strong>game-repl()</strong>:<br>    game-print(game-eval(game-read (read-user-input()))<br>    game-repl()</pre><p>What can we infer from the above snippet?</p><ul><li>The function game-repl calls itself recursively, creating an infinite loop.</li><li>The expressions are evaluated in normal order, i.e. each function is called starting from the innermost to outermost, right to left.</li><li>Thus the order in which the functions are called is as follows:</li></ul><pre><strong>1. read-user-input()</strong> : Takes what the user types in the command line interface and passes it to the next function game-read as an argument.</pre><pre><strong>2. game-read()</strong>: Takes the string of text typed by user, and converts it into a command understandable by the next function, game-eval.</pre><pre><strong>3. game-eval()</strong>: Takes the command, checks if it is an allowed one, and executes it. This function returns whatever was returned by the command execution.</pre><pre><strong>4. game-print()</strong>: Takes the result of the command execution, and prints it in a readable format for the player.</pre><pre><strong>5. game-repl()</strong>: The REPL is called again, and the process repeats, starting from 1.</pre><p>After reading what each function does, you’ve probably been thinking that most of the functionality of the game lies in the game-eval function.</p><p>You’re absolutely right. The ‘commands’ executed by game-eval handle all of the game’s defined logic. In fact, at the program level, <strong><em>game-eval()</em></strong><em> itself takes the user input as an argument, responds to the user input and returns the output to show the user</em>. The other components of the REPL merely provide an interface with the user — <strong><em>game-read()</em></strong><em> converts user input into the program’s language</em>, and <strong><em>game-print()</em></strong><em> converts the program’s language into the user output</em>, in the user’s language.</p><h4><strong>File structure:</strong></h4><p><em>How does game-eval process user input and return the required output? </em>It uses various other functions and variables to make this possible. These functions and variables are organised as follows into files:</p><ul><li><strong><em>Game UI Actions</em></strong><em>:</em> (game-uis.lisp)<br>user inputs passed to eval are commands, which are various functions defined in this file. Each of these functions in some way or the other, change the game state, which are variables defined in the next file.</li><li><strong><em>Game State</em></strong><em>:</em> (game-state.lisp)<br>This file contains a set of mutable data-structures that represent the state of the game — in our case, where the player is in the world, different aspects of the character’s state such as objects he has collected. The game state thus includes the things that change as the player plays the game. In terms of software, this game state consists merely of variable definitions.</li><li><strong><em>Game Initialization Data</em></strong><em>: </em>(game-data.lisp)<br>This file consists of game initialization data. The different places and their descriptions, the objects there and the different pathways.</li><li><strong><em>Scene Descriptors</em></strong><em>:</em> (scene-descriptors.lisp)<br>This file contains functions that process the variables in game-data along with game-state to describe the scene that can be shown to the user for his/her understanding — i.e. to re-render the game world.</li></ul><h4><strong>What do these files actually contain?</strong></h4><p>If you haven’t already looked through the GitHub code, you may be longing for a more concrete idea of what each file contains and how the functions are implemented.</p><p><strong>Game UI Actions:<br></strong>This file contains a function for each of the actions that a user can carry out. In our simple version of the game, these are also the commands that the user can type to interact with the game world.</p><pre><strong>1. look</strong>: describe the current place<br><strong>2. walk</strong> <em>direction</em>: walk and follow a path (east, west etc.)<br><strong>3. pickup </strong><em>object</em>: pickup an object in the current place<em><br></em><strong>4. inventory</strong>: describe what the player is carrying<br><strong>5.</strong> <strong>weld</strong> subject object: weld subject to the object if allowed<br><strong>6. dunk</strong> subject object: dunk subject into object if allowed. <br><strong>7. splash</strong> subject object: splash subject onto onto object</pre><p>It also contains a variable called <strong>*allowed-commands*</strong>, which contains a list of the commands that the user is allowed to execute, i.e. the above commands. This is for security purposes, to ensure that the user cannot type other commands in the REPL.</p><p><strong>Game State:</strong></p><pre><strong>* location</strong>: where the player is (i.e. which node)<br>* <strong>inventory</strong>: what the player is carrying (i.e. objects)</pre><p><strong>Game Data:<br></strong>We use a space graph to represent the places in the game, i.e. a graph consisting of nodes (places) and edges (pathways). We use a Lisp a-list (association list) for these data.</p><pre><strong>* nodes</strong>: contains the places and their description as an a-list<strong><br>* edges</strong>: contains the description of pathways to and from nodes<strong><br>* objects</strong>: contains the list of objects that can be picked up<strong><br>* object-locations</strong>: contains the objects and their location (i.e. which node)</pre><p><strong>Scene Descriptors:<br></strong>These functions describe the data in game-data and game-state in human-readable formats, i.e. as a sentence in the English language as output in the user’s console.</p><pre><strong>* describe-location</strong>: describes any given node in readable form.<strong><br>* describe-paths</strong>: describes a given edge (path) with direction and point of entry<strong><br>* describe-objects</strong>: describes the objects and where they lie, given a node/place<strong><br>* objects-at</strong></pre><h4><strong>How do these components work together?</strong></h4><ol><li>game-eval calls one of the the <em>game UI action functions</em> which look at the <em>current state of the game </em>(game-state variables) and the <em>game world </em>(game-data variables) and<em> change it</em> in the required ways.</li><li>Then the <em>scene descriptors</em> are called which <em>describe the game state and game world</em> in human-readable form and pass it to <em>game-print</em> to be displayed as output</li></ol><p>Here’s a flowchart to visualize that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*zaXDgPB8TEKXFdLZ4jyFvg.png" /><figcaption>Flowchart representing how the different components work together</figcaption></figure><h4><strong>Insights and Extensions</strong></h4><p>Now that you have an idea how this game was structured, let’s look at interesting features and functions that we can add. The best part of having a good architecture is that it can be modified or extended easily without changing the overall structure or schema. In fact, I find architecting solutions exciting for the very reason that you get to design structures which make it easy to <em>assimilate </em>new features without having to redesign the structure to <em>accommodate</em> these features.</p><p>Obviously, adding new actions in the game is as trivial as adding a new a game UI action, and corresponding variables in game-state if required. The game world can be extended by changing the game-data variables.</p><p>Here are a few other interesting things you could do to make the software structure of this game interesting.</p><ul><li><strong>Diffing scene graph</strong><br>In games where the output consists of many visual elements whose rendering is slow (for e.g.: using SVG in the DOM to render graphics), then one can make the rendering faster by rendering only the things that have changed and leave the things that haven’t changed as is. This algorithm that finds what has changed is called a <em>diffing algorithm</em>, and would have to be reasonably fast. The diffing is done on a<em> scene graph</em> (or a virtual DOM in the case of SVG via DOM) which is a data structure that represents the visual state of the screen.</li><li><strong>Functional game</strong><br>As described in detail in my article on Modelling Worlds using FP, one could easily figure out how to design games that do not change state, or have side-effects. Thus, like the game-data variables, even the game-state variables would then become immutable. Multiple performance optimizations can be carried out to make this somewhat practical.</li><li><strong>New forms of user input</strong><br>What if we wanted to use the arrow keys instead of typing <em>walk east</em>, <em>walk north</em> and so on? We could easily modify game-read to listen for such user input and convert it into these commands that game-eval understands.</li><li><strong>Metaprogramming<br></strong>In the original version of the book in which this game appeared, the author teaches you to create a <em>macro </em>that allows you to add game functions during run-time! I.e., Game UI Actions like weld, dunk and splash are created by a ‘function’ while the game runs —a runtime feature generation of sorts.</li></ul><h4>Concluding Remarks</h4><p>Building this basic game taught me a lot about Lisp and architectures, very early on in my programming experience.</p><p>I started building this game while learning Lisp using Conrad L. Barski’s book Land of Lisp, which taught me about various programming paradigms apart from the CLISP syntax by having me code games, web servers and other applications in CLISP in a hands-on manner. I found the book extremely fun to work through.</p><p>Thus, the rights to the concept of this game (The Wizard’s Adventure) lies entirely in the hands of the author of that book, and over time I have simply extended it to learn about the software structure and game mechanics of a text-game.</p><p><strong><em>The author is an education enthusiast, and works as a Software Engineer. His other publications include:</em></strong></p><ul><li><a href="https://medium.com/the-software-firehose">The Software Firehose</a></li><li><a href="https://medium.com/hacking-learning">Hacking Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e0dc4d49812" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-software-firehose/architecting-a-text-adventure-game-2e0dc4d49812">A Text Adventure Game in CLISP</a> was originally published in <a href="https://medium.com/the-software-firehose">The Software Firehose</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to choose a table index for your SQL database]]></title>
            <link>https://medium.com/the-software-firehose/how-to-choose-a-table-index-for-your-sql-database-d47715a35f34?source=rss----bfe766a2c671---4</link>
            <guid isPermaLink="false">https://medium.com/p/d47715a35f34</guid>
            <category><![CDATA[practical-tips]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Shaaz Ahmed]]></dc:creator>
            <pubDate>Thu, 22 Mar 2018 19:49:01 GMT</pubDate>
            <atom:updated>2018-03-23T17:31:56.479Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*ynCpkAOf5ONJzbQoX1Wngg.jpeg" /></figure><p>A lot of people seem to be working with SQL databases without much technical training recently, and I figured many such people might want some help optimizing their queries for read, write, and update operations on the database without having to pick up a lot of theory.</p><p>This one’s for the absolute beginners to indexes. So if you know how a database index works, you probably shouldn’t be reading any further.</p><p>RDBMS are very robust and powerful systems if used in the right way, even for the amounts which many <a href="https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html">people wrongly consider ‘big’ data</a>. Choosing the right table index for your database is one of the easiest ways to optimize database performance, and it’s also the primary database optimization that a developer should be concerned with.</p><p><em>The person who can optimize queries by creating an index is the person who knows what the queries are.</em></p><p>This also means that if you want to future-proof your work, you should also be able to <em>anticipate what future queries your application will be making to the database</em>.</p><p>If you want to know how database indexes work in detail but without all the other fluff, <a href="http://use-the-index-luke.com/">www.use-the-index-luke.com</a> is an excellent free resource. It won’t do justice to that resource if I tried to cover the details that it explains so well, so check it out if you’re curious (and you can stop reading this post right away).</p><h4>The Levels of Optimization</h4><p>Before we start, it helps to know that you can always optimize performance at many levels, and even with your SQL database, that can happen at at least two levels:</p><ol><li><strong>Application Level</strong>: By examining exactly what data you need for your application, and requesting only that data without any overhead or extra data.</li><li><strong>Query Level</strong>: By <em>choosing better indexes for your most common queries</em>, and sometimes by <em>restructuring your query in more efficient ways</em> to use existing indexes or a simpler operation.</li></ol><p>Throughout this quick ‘tutorial’, we’ll be focusing entirely on <em>query level optimization using indexes</em>.</p><h4>What is an index?</h4><p>Let’s try to understand an index using an analogy. Say we have a table called ‘<strong>customers’ </strong>in your database, with their names and phone numbers.</p><p>To build our analogy, let’s imagine an equivalent solution: we have a book/register containing a large list of customers and their phone numbers with the respective row number.<br>For e.g., a book with hundreds of thousands of entries in this format:</p><pre>...  ...                      ...<br>678. Hunter, Alex             9123220322<br>679. Rhodes, William          9873344323<br>680. Mata, Juan               9234232234<br>681. Cardoza, Johann          9858389233<br>...  ...                      ...</pre><p>Imagine, for some reason, that our business’s service department has to often find out the customer’s name given his phone number. You can imagine how hard a task this is, having to go through each number from beginning to end sequentially and check if it matches that number.</p><p>You’re probably thinking: why don’t we just write down all these number-name pairs in ascending order of the numbers, i.e. why don’t we just <strong>sort </strong>the phone numbers?</p><p>The issue with that is quite evident: writing new data becomes impossible. Why? Because the physical order is dependent on when the data is entered, so we’d have to change the physical order every time new data is entered. (It’s also interesting to note that by simply entering data as and when it comes, the row number is automatically sorted by time of entry.)</p><p>So we try a different solution.</p><p>We buy another book/register, and list down all the phone numbers in ascending order along with the row number of that data in our original customer book, leaving enough space to add new data in the future if required. This is your ‘index’ (yes, just like a textbook’s index at the back).</p><pre>...  ...                      ...<br>9123220322                    187<br>9123224195                    892<br>9123298322                    14<br>...  ...                      ...</pre><p>Now it becomes very easy for us to find the customer name by using both books. This may be slower in some cases (e.g. for small tables where sequential checking is faster due to the overhead of keeping an index), but on average this will definitely be faster.</p><p>Now, I’d like you to note that the part about ‘leaving enough space’ is not technically accurate in terms of how it’s implemented with databases, and was used purely for lack of a better analogy.</p><p>Most disk-based databases use a B-tree (or balanced tree) data structure to store indexes — this allows quicker searching of data on average. Visit <a href="http://use-the-index-luke.com/sql/anatomy/the-tree">http://use-the-index-luke.com/sql/anatomy/the-tree</a> to learn more.</p><p>When you define an index on the column of a database table, the database management system (DBMS, like MySQL, Postgres etc.) is essentially creating another ‘table’ with the data of that particular column sorted, allowing quicker search time. How you create the index is specific to your DBMS. Note that the overhead introduced for maintaining indexes may not be worth it for tables with a small number of rows.</p><p>Now, getting to the point of this article:</p><h4><strong>So, which column of the table should I define the index on?</strong></h4><p>Make a list of all the queries your application is making to the particular table, and figure out which column is the most commonly used WHERE clause or JOIN ON clause. That’s the column you should define your index on.</p><p>For e.g., if WHERE <strong><em>phone</em></strong> = 923123121 <strong>or</strong> JOIN ON customers.<strong><em>phone</em></strong> = foo.bar are cases that appear often in your query, then you should (in most cases) define your index on the ‘phone’ column.</p><p><strong>Multi-column Indexes</strong></p><p>When your WHERE (or JOIN ON) clauses contain more than one condition, involving more than one column, then you could define an index on two columns. For e.g. if you define an index on the (<strong>age</strong>, <strong>phoneno)</strong> columns of a table, then the index will be stored in order of sorted age, and for each age, the phone-number will be sorted. This quickens up two types of queries:</p><ul><li>ones with ‘WHERE age = xx AND phoneno = xxxxxxxxx’ and</li><li>one with just ‘WHERE age = xx’</li></ul><p>However, a query with WHERE phoneno = xxxxxxxxx would still be slow, because we defined the index on (age, phoneno), and not (phoneno, age). Thus, the <em>order of definition of multicolumn indexes matters.</em></p><p><strong>Primary Keys</strong></p><p>Primary keys are simply columns with <br>- an index defined on them, <br>- a UNIQUE constraint (not duplicates) and,<br>- a NOT NULL constraint</p><h4>Concluding Remarks</h4><p>A few things I’d like you to note:</p><ul><li>Time your queries. High performance = low query time, from an application developer’s perspective.</li><li>The size/optimization tradeoff: As mentioned earlier, the overhead of introducing indexes will cause your queries to be <em>slower </em>if your table isn’t big enough.</li><li>Trade-offs: With every optimization, there can be various ‘gotchas’ — optimizing one type of query can slow down the other and so on. If you want to understand the nuances, or even just figure out why your queries are slow, learn more details from a resource like the one mentioned at the end of this post.</li></ul><h4>Performance Gains: A Real Example</h4><p>I’ll narrate an actual example from personal experience when database optimization helped us. For our product Engaze, once we had developed a barebones/minimal functioning user interface and backend, we realized that requests to our data API was taking quite a bit of time.</p><p>Looking at the timings for requests in the Chrome console, one request revealed itself as the main culprit, the ActivitySuggestions request— it was fetching a list of activities for the user after some heavy data analytics.</p><p>To quantify things, ActivitySuggestions was taking 18 seconds! That’s a lot of time, honestly.</p><p>Running the Flask profiler, we figured more than 95% of that time was spent in one particular function. Guess what? That function had 6 SQL queries to our database that was slowing things down.</p><p>My colleague Abhit performed the first (application) level of optimization, by combining information in 2 queries neatly into one better query. Now, the ActivitySuggestions request took only 8 seconds, a significant improvement!</p><p>I continued to examine what was wrong, and realized that two of the remaining queries took ~2.87 seconds to complete. Well, I put all the queries in a list, looked at their WHERE clauses, and it became quite obvious to me what the index had to be for our table of about ~30 million rows (then). So I defined a multi-column index for that table.</p><p>Can you guess how much time the 8 second request took now? 700ms! Yes, 0.7s. The 2.87 second SQL query itself took only 0.13 seconds. That’s at least a 50 times increase purely from SQL query optimization!</p><p><em>If you’re still curious about how indexes work, I recommend you read the articles on </em><a href="http://use-the-index-luke.com./"><em>http://use-the-index-luke.com.</em></a><em> It’s quite a lot of fun if you’re just a beginner using databases and care only about the theory that has relevance for your performance-optimization needs as an application developer.</em></p><p><strong><em>The author is an education enthusiast, an apprentice flaneur and works as a Software Engineer at </em></strong><a href="http://nilenso.com/"><strong><em>Nilenso Software</em></strong></a><strong><em>, a boutique software consultancy and employee-owned software cooperative. His other publications include:</em></strong></p><ul><li><a href="https://medium.com/the-software-firehose">The Software Firehose</a></li><li><a href="https://medium.com/hacking-learning">Hacking Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d47715a35f34" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-software-firehose/how-to-choose-a-table-index-for-your-sql-database-d47715a35f34">How to choose a table index for your SQL database</a> was originally published in <a href="https://medium.com/the-software-firehose">The Software Firehose</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Diffie-Hellman Key Exchange Protocol, simplified]]></title>
            <link>https://medium.com/the-software-firehose/how-to-share-a-secret-key-part-1-7c3ea83ab3bb?source=rss----bfe766a2c671---4</link>
            <guid isPermaLink="false">https://medium.com/p/7c3ea83ab3bb</guid>
            <category><![CDATA[end-to-end-encryption]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[encryption]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[Shaaz Ahmed]]></dc:creator>
            <pubDate>Thu, 22 Mar 2018 19:33:48 GMT</pubDate>
            <atom:updated>2019-01-19T21:36:55.889Z</atom:updated>
            <content:encoded><![CDATA[<h4>… or: how to share a secret.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*7op9EdUMWz42CBX6Qu-nfw.png" /><figcaption>The Diffie-Hellman family of protocols is widely used to make insecure channels secure</figcaption></figure><p>The Diffie-Hellman key exchange has been receiving a lot more attention since its use for implementing end-to-end encryption on WhatsApp, using the Signal Protocol. One of the components of the Signal Protocol is a slightly more advanced version of the Diffie-Hellman concept we will discuss below, called the X3DH or Triple-Extended Diffie-Hellman. It will be significantly easier to understand X3DH in the next post once you’ve understood the basic underlying concepts as described below.</p><p>Many schemes of encryption of data between two parties is based on the idea that both parties know a shared secret key that no one else knows, and both of them can use this key to encrypt messages before sending them and decrypt messages from the other party. Such algorithms which use the same key for encryption of plain-text and the decryption of cipher-text (i.e. the encrypted text) are called symmetric-key algorithms in cryptography.</p><p>However, something about this puzzled me initially. <strong><em>How did these two parties get the shared key in the first place?</em></strong> Surely, they can’t just send it to each other over the network — the first time one of them sends it to the other, the message is not encrypted and hence anyone listening in to the network can intercept that message and get hold of the key. This looks like a chicken-and-egg problem — how can you use a symmetric-key encryption scheme in a network that’s not already encrypted? This is where the Diffie-Hellman exchange comes into play.</p><p>Before discussing the Diffie-Hellman exchange, I’ll summarize the approach this post takes. This post will be split into two parts — in the first part, we will discuss the basic idea behind the Diffie-Hellman key exchange in an extremely simple manner that is accessible to people without any previous knowledge of cryptography (e.g.: me when I started writing this). In the second part, we will discuss more practical examples, including the more sophisticated Triple-Extended Diffie-Hellman (X3DH) protocol.</p><h3>A Simplified Explanation: Creating &gt; Sharing</h3><p>The Diffie-Hellman exchange is a key agreement protocol — a protocol that is specifically designed to help you create a secret key that both parties have, but don’t need to send across the network to share it. How is that possible? Thanks to a few neat mathematical tricks, both parties can generate a secret key using their own secrets without ever explicitly sending it to each other over the network.</p><h4>Example</h4><p>Let’s take a trivial example that outlines the idea behind the protocol, with many practical details omitted.</p><p>You want to send a message to your friend Alice. However, you don’t want anyone apart from Alice to know what the contents of the message are. You’re using the <a href="https://en.wikipedia.org/wiki/Caesar_cipher">Caesar cipher</a>, a simple encryption technique in which each letter in your address is replaced by a letter that is a fixed number of positions down in the alphabet. This fixed number of positions is the shared key — if Alice has this key, she can decrypt your message. (<em>Note: the Caesar cipher is an outdated encryption scheme, and can be broken very easily — we only consider it as a simple example</em>)</p><p>Instead of sending a key over to Alice so that you two have a shared key, follow the following steps.</p><p><strong>Step 1: <em>Pick two prime numbers and send them to Alice</em></strong><br>First, you pick two prime numbers <strong>g</strong> and <strong>p, </strong>and send it to Alice. For example, g = 3, and p = 19.</p><p><strong>Step 2: <em>Alice computes A using her own secret key, and sends A to you</em></strong><br>To create the first part of the key, Alice picks her own secret number <strong>a</strong> (say, a = 4) and does the following computation:</p><p>g^a <em>mod</em> p <strong>=</strong> 3⁴ <em>mod</em> 19= 81 <em>mod</em> 19= 5</p><p>Alice then sends this number <strong>A</strong> (= 5) across to you. (we chose to label it <strong>A </strong>because it’s derived from Alice’s secret <strong>a</strong>)</p><blockquote>What is the mod operator? In the operation <strong>a </strong>mod <strong>b</strong>, the modulo operator finds the remainder after division of <strong>a</strong> by <strong>b</strong>. E.g. <strong>10</strong> mod <strong>3</strong> = <strong>1</strong>.</blockquote><p><strong>Step 3: <em>You compute B using your own secret key, and send B to Alice</em></strong><br>Meanwhile, you pick your own secret number <strong>b </strong>(say, b =13)<strong>, </strong>and do a similar computation:</p><p>g^b <em>mod</em> p = 3¹³ <em>mod </em>19 = 1594323 <em>mod</em> 19 = 14</p><p>You then send this number <strong>B </strong>(= 14) across to Alice.</p><p>Now, you have the number 5 that Alice sent you, and Alice has the number 14 that you sent her.</p><p><strong>Step 4: Alice computes the shared key</strong></p><p>Alice now uses her secret <strong>a </strong>(= 4)<strong> </strong>and the number you sent her <strong>B </strong>(= 14)to<strong> </strong>compute the following:</p><p>14^a <em>mod</em> p = 14⁴ <em>mod</em> 19 = 38416 <em>mod</em> 19 = 17</p><p><strong>Step 5: You compute the shared key</strong></p><p>Similarly, you use your secret <strong>b (=13) </strong>and<strong> </strong>the number Alice sent you <strong>A </strong>(= 5) to compute:</p><p>5^b <em>mod</em> p = 5¹³ <em>mod</em> 19 = 1220703125 <em>mod</em> 19 = 17</p><p><strong>Wow, both of you now have the same key — 17!</strong> And none of you had to send it to each other explicitly. Now, both of you can shift all the letters in your message by 17 letters (since we’re using the Caesar cipher) and encrypt your communication with each other, and shift it back by 17 letters to decipher it.</p><p>Now, there are probably two burning questions in your head right now:</p><p>i) How did this happen? Will both numbers always be the same?<br>ii) Why couldn’t someone intercept the numbers that we sent across the network (g, p, A and B) and simply use them to obtain the key?</p><h4>Modular Exponentiation: how did this happen?</h4><p>Let’s tackle the first question: how did this happen? The <em>mod </em>(modulo) operation has a nifty little property when it comes to exponentiation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*7eecc0DSP4o-dlDDcRrR7Q.png" /></figure><p>where <strong>a </strong>and <strong>b </strong>are the ‘secrets’ we used earlier. What does this imply? This implies that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*FtfnbEwy4ZE7bXERhVFKbg.png" /><figcaption>a and b are individual secrets, but the final result is a shared secret</figcaption></figure><p>Hence, you don’t need to know what <strong>a</strong> is and Alice doesn’t need to know what <strong>b</strong> is, but both of you know the value of the number <strong>g^(a*b) <em>mod</em> p.</strong></p><h4>The Discrete Logarithm: why is this difficult to break?</h4><p>Now, let’s get to the more important question: why can’t someone simply intercept the numbers we send across the network and use it compute the final key? The difficulty in reverse engineering the shared secret key lies in the difficulty to compute the discrete logarithm. What is the discrete logarithm?</p><p>The regular logarithm in mathematics is the inverse operation of exponentiation — if 3² = 9, then the log of 9 to the base 3, is 2. The discrete logarithm is very close to this logarithm. When we compute <em>4² mod 3 = 1</em>, how do we find the value 1 (the shared ‘secret’), if we only knew the values of 4 and 3 (which are the only two numbers we can get from snooping on the communication)? I.e., how do we find <em>y</em> in this equation<em>:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/1*vP3dneJcRCG8RHxvAEKs0Q.png" /></figure><p>Remember, the modulo operator is the remainder after division of 4^x by 3. So, if <strong>n</strong> was the quotient of that division,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*4To0rUUOyQbPTukZuvSELA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*eLDPXCXWLnrqIt4KJhHirw.png" /></figure><p>Now, an attacker may find out <strong>y </strong>by listening in to our communications (remember, Alice sends <strong>A</strong> to you and you send <strong>B</strong> to Alice — these are equivalent to <strong>y</strong>). Even if we assumed that n = 1, there’s a range of possibilities for the values of x and y that will satisfy this equation. <strong>However, we require x to be an integer — i.e. a discrete number.</strong> The logarithm which has a discrete value (an integer, and not a decimal number) is called a discrete logarithm, and this makes the problem a lot harder, except in some specific cases. If we carefully avoid the special cases where it is easy to compute the discrete logarithm, there is no general and efficient way to compute them and makes them suited for use in cryptography.</p><h4>Additional notes</h4><p>The above explained scheme has many weaknesses, some of which we will discuss while introducing the X3DH protocol in Part 2. Note that, X3DH and more modern implementations don’t use the modulo operator, but operations that are even more difficult to compute the inverse of — such as the elliptic curves Curve25519 and Curve448. Also note that, although we’ve used <strong>g</strong> and <strong>p </strong>as two prime numbers, they are usually prime generators, or functions that can generate prime numbers.</p><h4>Bibliography</h4><p>[1] <a href="https://signal.org/docs/specifications/x3dh/x3dh.pdf">The X3DH Key Agreement Protocol</a>, <em>Signal</em><br>[2] <a href="https://security.stackexchange.com/questions/45963/diffie-hellman-key-exchange-in-plain-english">“Diffie Hellman Key Exchange” in plain English</a>, <em>Stackexchange<br></em>[3] <a href="https://www.mathjax.org/#demo">MathJax Display Engine</a><br>[4] <a href="https://signal.org/docs/specifications/doubleratchet/doubleratchet.pdf">The Double Ratchet Algorithm</a>, <em>Signal</em></p><ul><li><a href="https://medium.com/the-software-firehose">The Software Firehose</a></li><li><a href="https://medium.com/hacking-learning">Hacking Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c3ea83ab3bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-software-firehose/how-to-share-a-secret-key-part-1-7c3ea83ab3bb">The Diffie-Hellman Key Exchange Protocol, simplified</a> was originally published in <a href="https://medium.com/the-software-firehose">The Software Firehose</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>