<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nishanth Shetty (@nsp) on Medium]]></title>
        <description><![CDATA[Stories by Nishanth Shetty (@nsp) on Medium]]></description>
        <link>https://medium.com/@nishanthspshetty?source=rss-f5de13ae96b2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2X6-mSe7v_sKTvQAdNnWgg.jpeg</url>
            <title>Stories by Nishanth Shetty (@nsp) on Medium</title>
            <link>https://medium.com/@nishanthspshetty?source=rss-f5de13ae96b2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:06:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nishanthspshetty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Managing multiple EKS cluster with different AWS accounts]]></title>
            <link>https://medium.com/@nishanthspshetty/managing-multiple-eks-cluster-with-different-aws-accounts-8c793adcabf0?source=rss-f5de13ae96b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c793adcabf0</guid>
            <category><![CDATA[aws-cli]]></category>
            <category><![CDATA[elastickubernetesservice]]></category>
            <category><![CDATA[kubectl]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Nishanth Shetty (@nsp)]]></dc:creator>
            <pubDate>Wed, 06 Sep 2023 16:59:59 GMT</pubDate>
            <atom:updated>2023-12-24T07:20:13.486Z</atom:updated>
            <content:encoded><![CDATA[<p>At my current work, I need to use different clusters with different aws account associated with. Company will use dev and prod clusters and switch between the two using kube context with the help of kubectl and access the cluster.</p><p>But we have a two different account setup, which requires us to configure aws cli to required account before switching the context for k8s.</p><p>This is what we did more like what my colleague was using, a simple use of bash script and aws profile.</p><ol><li>Use aws profile option to set up multiple account, we will do for dev and prod</li></ol><pre>aws configure --profile dev # enter your account id and secrets<br>aws configure --profile prod</pre><p>2. Create a shell script(command) to switch the profile<br> <br>For switching to dev</p><pre>#!/bin/bash<br>export AWS_PROFILE=dev<br>aws eks update-kubeconfig --name dev-cluster-name --region ap-south-5<br>eksctl get cluster<br><br>## save this as `eksdev` in $PATH and run the script whenever needed.</pre><p>For switching to prod</p><pre>#!/bin/bash<br>export AWS_PROFILE=prod<br>aws eks update-kubeconfig --name prod-cluster-name --region ap-south-5<br>eksctl get cluster<br><br><br>## save this as `eksprod` in $PATH and run the script whenever needed.</pre><p>3. In kube config add environment variable called AWS_PROFILE as follows</p><pre>- name: your-cluster-arn<br>user:<br>  exec: exec command and its args<br>    ....<br>    command: aws<br>    env:<br>     - name: AWS_PROFILE<br>       value: prod</pre><p>4. Run the script to switch the context</p><pre>$ eksdev</pre><p>3. Now you can use `kubectl` or `k9s` to query the cluster</p><p>Reference:</p><ol><li><a href="https://www.linkedin.com/pulse/aws-cli-multiple-profiles-bachar-hakam/">https://www.linkedin.com/pulse/aws-cli-multiple-profiles-bachar-hakam/</a></li><li><a href="https://docs.aws.amazon.com/cli/latest/reference/configure/#configure">https://docs.aws.amazon.com/cli/latest/reference/configure/#configure</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c793adcabf0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lignum: distributed message queue]]></title>
            <link>https://blog.devgenius.io/lignum-distributed-message-queue-403d38548dbe?source=rss-f5de13ae96b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/403d38548dbe</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Nishanth Shetty (@nsp)]]></dc:creator>
            <pubDate>Thu, 24 Nov 2022 10:30:33 GMT</pubDate>
            <atom:updated>2022-11-25T16:00:40.065Z</atom:updated>
            <content:encoded><![CDATA[<h3>Lignum : distributed message queue</h3><p>In between all the WFH fuzz and isolated situation, I set out myself to look into distributed locks using Consul, the simple test program soon became a path to building a distributed message queue called <a href="https://github.com/NishanthSpShetty/lignum">Lignum</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6OjQv7lpEg7P4h3IqQZKw.jpeg" /><figcaption>Lignum overview</figcaption></figure><p>This is an introductory article to Lignum message queue.</p><p><strong>What is distributed system?</strong></p><p>There are many definitions out there, a simple google search will result in multiple answers. I can define distributed system as a collection of autonomous or individual program or systems which are working towards a common goal.</p><p>Meaning, multiple systems networked together by various means of communication to serve a common purpose. It could be systems where one ask for web page and one serves web pages. For an end user these systems may appear as one, but there could be hundreds of such systems, who communicates with each other to do the task.</p><p><strong>What is message queue?</strong></p><p>System which provides queue data structure for other service to read and write messages in FIFO order.<br>Queue allows producer service to write message and consumer to consume asynchronously. Producer and consumer need not to be aware of each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZJzr7Loa3Ad7oxF1S-ILw.jpeg" /><figcaption>message queue</figcaption></figure><p>Two basic characteristics of the message queue are</p><ol><li>Topic<br>Messages are sent to the named buffer, these are called topic. We produce and consume messages from the topic.</li><li>Message<br>This is the actual message we write to topic for others to consume.</li></ol><p>Example: Apache Kafka, RabbitMQ, Celery.</p><p><strong>What is distributed message queue?</strong></p><p>When we build a message queue which are hosted in multiple nodes, collectively working together to serve as single entity for other services.</p><p>There will be multiple queue hosting user messages to provide</p><ol><li>Reliability and high availability</li><li>Fault tolerance</li><li>Performance</li><li>Sharing read load with multiple nodes</li></ol><p><strong>Lignum</strong></p><p><a href="https://github.com/NishanthSpShetty/lignum">Lignum</a> was started as a test program to learn distributed locking using <a href="https://developer.hashicorp.com/consul/docs/dynamic-app-config/sessions">Consul session lock</a>, which started to pivot towards message queue as I was working on Kafka at that time. <br>This is written in Go. I did start it with test driven development in mind but soon it ended up being a mess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*_xry6EEE9dLNkq1Bs2x9Pg.png" /><figcaption>Lignum Architecture</figcaption></figure><p>Lignum isn’t a full fledged message queue, its a work in progress. You can</p><ul><li>Send messages to topic</li><li>Consume messages from topic</li><li>Message is replicated to all nodes, meaning any node can be used to read message from the topic</li><li>Persist messages on disk</li></ul><p><strong>Lignum cluster</strong></p><ul><li>Lignum can operate in cluster mode without needing any special setup</li><li>Cluster management is facilitated by Consul with its session locks</li><li>Each node will connect to Consul to get the leader. If no leader is found, one of the node will be elected as the leader</li><li>All other node will register itself as follower to the leader</li><li>Message sent to the leader will be replicated to the follower node</li><li>It has two modes of replication <br> a. WAL replication<br> b. Live replication</li><li>If the leader goes down, any one of the follower gets elected as the leader</li></ul><p><strong>Replication</strong></p><p>Currently lignum implements two modes of replication strategy.</p><ol><li>WAL replication</li><li>Live replication.</li></ol><p>By default, WAL replication is chosen.</p><p><strong>WAL replication</strong></p><p>Log file gets created for each topic when the message spills over buffer size defined in config(message.initial-szie-per-topic). Lignum creates WAL file(.qwal) for each topic. When the message count reaches the buffer size, this file gets flushed to disk as log file. Lignum frequently watches for these log file creation and sends over to the followers. All followers get synced up periodically with the leader, providing eventual/delayed consistency.</p><p><strong>Live replication</strong></p><p>This is a streaming replication, when the follower is already registered with the leader, we don’t need to worry about syncing the messages if the topic created after. We can just forward the messages received in the leader as is to follower. In this strategy, messages get sent to registered follower immediately. This way Lignum can provide strong consistency.</p><p><strong>How to run Lignum</strong></p><ul><li>Prerequisite<br> — Consul <br> You can use Dockerfile as specified in <a href="https://github.com/NishanthSpShetty/lignum#requirement">README</a></li><li>Clone the repository</li></ul><pre>https://github.com/NishanthSpShetty/lignum.git</pre><ul><li>build it make build</li><li>update config.yml , you can refer <a href="https://github.com/NishanthSpShetty/lignum/blob/master/doc/config.md">this documentation</a> for more details on configs.</li><li>run make run</li></ul><p>This will run a single node, elects itself as leader. You can write and read message from this now.</p><ul><li>Send message</li></ul><pre>curl --request POST \<br>  --url http://localhost:8080/api/message \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>   &quot;topic&quot;: &quot;test&quot;,<br>   &quot;message&quot;:&quot;this is a test message&quot;<br>   }&#39;</pre><ul><li>Read message</li></ul><pre>curl --request GET \<br>  --url http://localhost:8080/api/message \<br>  --header &#39;Content-Type: application/json&#39; \<br>  --data &#39;{<br>   &quot;topic&quot;: &quot;test&quot;,<br>   &quot;from&quot;: 0,<br>   &quot;to&quot;: 3<br>   }&#39;</pre><blockquote>NOTE : lignum doesn’t track consumers, so as of now we need to provide the offset to read messages.</blockquote><p>Messages starts with the offset 0. In the above example, the messages will be fetched from offset 0 to offset 2, i.e. [0,3). You will get 3 messages from the system if it has.</p><p>What delivery guarantee it offers?<br>As the messages are persisted and consumer can read any offset at anytime, the delivery guarantee lies with the consumer. The Lignum guarantees message ordering.</p><p>Lets add node to the existing cluster</p><ul><li>Change the config as per your host.</li><li>If running on same host as previous run, then change the port, data directory and replication port values in config file.</li><li>Run make run</li></ul><p>This will bring up another node, registers itself with the leader node. As soon as the follower is registered, any topic presents in the leader node will be replicated with all the log files it has as explained in WAL replication. Now you can read the messages from the follower node as well.</p><p>This is a short introduction to Lignum. I will come up with a separate document on its internals and development details in the future.</p><p>Code is in Github :<a href="https://github.com/NishanthSpShetty/lignum">Lignum</a>.</p><p>Checkout and provide feedback, contributions are welcome.</p><p>Checkout my Github profile for other awesome projects I’m working on</p><p><a href="https://github.com/NishanthSpShetty">NishanthSpShetty - Overview</a></p><p>Connect with me on LinkedIn</p><p><a href="https://www.linkedin.com/in/nishanthspshetty/">https://www.linkedin.com/in/nishanthspshetty/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=403d38548dbe" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/lignum-distributed-message-queue-403d38548dbe">Lignum: distributed message queue</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[neovim ❤ elixir]]></title>
            <link>https://medium.com/@nishanthspshetty/neovim-elixir-b2b3904cc9c2?source=rss-f5de13ae96b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2b3904cc9c2</guid>
            <category><![CDATA[code]]></category>
            <category><![CDATA[neovim]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[elixir]]></category>
            <dc:creator><![CDATA[Nishanth Shetty (@nsp)]]></dc:creator>
            <pubDate>Mon, 05 Sep 2022 17:03:01 GMT</pubDate>
            <atom:updated>2022-09-07T15:26:03.620Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Setting up elixir workflow with neovim ❤</strong></h4><p>I have recently started looking into elixir, just like everyone the first thing I set out to do is to get my neovim working for elixir with the `lsp` support. It wasn&#39;t straight forward per say.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*DKu16j1IVyEcO70Ucf1Q-Q.jpeg" /></figure><p>So, this is the rundown of how I setup my system with the all goodies of elixir in neovim.</p><p>Assuming that you have neovim setup and have a fair idea of how to setup any lsp servers.</p><blockquote>Skip the below section if you have elixir setup already</blockquote><p><strong>Installing asdf</strong></p><p><a href="https://asdf-vm.com/">asdf</a> is used to manage multiple versions of runtime using single cli tool. We will use this to install elixir and erlang-otp of a particular version on our system. Installation instructions can be found <a href="https://asdf-vm.com/guide/getting-started.html#_1-install-dependencies">here</a> . Once we have this setup, lets get to next step</p><p><strong>install elixir and erlang</strong></p><ul><li>Add erlang and elixir as plugin to asdf tool</li></ul><pre> asdf plugin add erlang<br> asdf plugin add elixir</pre><ul><li><strong>EDIT </strong>we need to install libssl and other dependent lib.</li></ul><pre>sudo apt install libssl-dev libncurses5-dev autoconf automake</pre><ul><li>Now install elixir and erlang</li></ul><pre>asdf install erlang 25.0.4<br>asdf install elixir 1.14.0-otp-25</pre><ul><li>Set the installed version as the current active version on the system</li></ul><pre>asdf global erlang 25.0.4<br>asdf global elixir 1.14.0-otp-25</pre><p>You can see the current installation by calling asdf current.Now that we have installed elixir, we are good to setup language server.</p><p><strong>Install elixir-ls</strong></p><p>We need to install the language server which can be found <a href="https://github.com/elixir-lsp/elixir-ls">i</a>n <a href="https://github.com/elixir-lsp/elixir-ls">elixir-lsp/elixir-ls</a>. We need to download the repo and build it.</p><ul><li>Clone the repo in home path/(your preferred path)</li></ul><pre>git clone <a href="/service/mailto:git@github.com">git@github.com</a>:elixir-lsp/elixir-ls.git ~/.elixir-ls</pre><ul><li>Get the project dependencies</li></ul><pre> mix deps.get</pre><ul><li>if any error occurred with deps, try updating</li></ul><pre>mix deps.update --all</pre><ul><li>compile and generate the release</li></ul><pre>mix compile &amp;&amp; mix elixir_ls.release -o release</pre><p>once we have the lsp built, lets get to the crucial part…</p><p><strong>Setup neovim with lsp plugin</strong></p><ul><li>Add lsp client plugin to neovim</li></ul><p>We will use the <a href="https://github.com/mhanberg/elixir.nvim">mhanberg/elixir.nvim</a> as lsp client for the elixir lsp server.</p><p>add the following section to your neovim config, resource the neovim and run :PluginInstall. I use <a href="https://github.com/VundleVim/Vundle.vim">vundle</a> to manage plugins so it would look like</p><pre>Plugin &#39;mhanberg/elixir.nvim&#39;</pre><p><em>use your preferred plugin manager to download.</em></p><ul><li>configure lsp client<br> Its easy to configure according to document as follows,</li></ul><pre>require(&quot;elixir&quot;).setup()</pre><p>The lsp client would install language server in the project repo and you should have the lsp working already in your project.</p><p>However, this didn’t work for me, so If you face the same issue we have to install elixir-lsmanually and connect the neovim to use our installed server.</p><p>To do this, we will modify our neovim config with the following.</p><pre>local elixir = require(&quot;elixir&quot;)<br>elixir.setup({<br> // here we specify the path we built our elixir-ls,<br> // here we setup this in home directory, <br> //make sure you are pointing to right path<br> cmd = { &quot;/home/nishanth/.elixir-ls/release/language_server.sh&quot; },<br> settings = elixir.settings({<br>  dialyzerEnabled = true,<br>  fetchDeps = false,<br>  enableTestLenses = false,<br>  suggestSpecs = false,<br> }),<br>})</pre><p>restart neovim and try again,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*dSygZurLA7Zy7CDgTwE1lw.png" /></figure><p>and, voila!!, once i did this, I got the neovim working with all the auto completion, suggestions, hints and all goodies of lsp.</p><p>You can check the lsp stat too by invoking :LspInfo</p><p>You can find the complete neovim config in my github <a href="https://github.com/NishanthSpShetty/dotfiles/tree/master/vimrc/nvim-lsp">here</a></p><p>Checkout my github profile for other awesome projects I’m working on</p><p><a href="https://github.com/NishanthSpShetty">NishanthSpShetty - Overview</a></p><ul><li>Lignum : Distributed message queue (akin kafka)<br> <a href="https://github.com/NishanthSpShetty/lignum">https://github.com/NishanthSpShetty/lignum</a></li><li>Rimit : implementation of different rate limiter algorithm in Rust <br><a href="https://github.com/NishanthSpShetty/rimit">https://github.com/NishanthSpShetty/rimit</a></li><li>Crust : C++ to Rust transpiler <a href="https://github.com/NishanthSpShetty/crust">https://github.com/NishanthSpShetty/crust</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2b3904cc9c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Two Years And A Thousand Learnings Later]]></title>
            <link>https://medium.com/gojekengineering/two-years-and-a-thousand-learnings-later-3ccdff6730c9?source=rss-f5de13ae96b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ccdff6730c9</guid>
            <category><![CDATA[gojek]]></category>
            <category><![CDATA[engineer]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[culture]]></category>
            <dc:creator><![CDATA[Nishanth Shetty (@nsp)]]></dc:creator>
            <pubDate>Tue, 04 Jan 2022 05:25:17 GMT</pubDate>
            <atom:updated>2022-01-04T05:25:17.835Z</atom:updated>
            <content:encoded><![CDATA[<h4>Our GoTroop talks about his Gojek journey so far and gives a peek into a day in his life.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gV1Sxwh58DrNR-gB0pyHOA.jpeg" /></figure><p>It’s been a little over two years since I joined Gojek and the journey has been nothing short of a dreamy experience — working along with talented individuals and experimenting with various ideas, I got a chance to try my hand at building new systems and features.</p><p>In this blog, you shall see my journey as a developer in Gojek.</p><h4><strong>The recruitment</strong></h4><p>The first I heard about the company was when the recruiter got in touch with me. I was impressed by the way they went through my profile with me. It’s one thing to be have done your research about the company. It’s an even better feeling when the org that wants to hire you does the same for you. I decided to take the job and I have been very happy with my decision ever since.</p><p>After my interaction with the recruiter, my thoughts were that if the recruiter was so involved with knowing about the tech before reaching out to candidates to make a good impression, then there must be something special about the company’s culture. I might not have been surprised if they spoke about Java, Python or any other widely used programming language. What jolted me was the comfort with which they spoke about Rust and the project I worked on using Rust.</p><p>I feel that an interview a two way process. Yes, an interview is a process where the company will assess you for your knowledge and see if you are fit for the role. But, it’s also a chance for a potential employee to know what kind of colleagues or culture they’re going to see in the team. I shot all questions I had in mind about the org, and was impressed with the answers I got. Everything went smoothly and I had an offer in my hand. I was officially a GoTroop. 🖖</p><h4><strong>Working at Gojek</strong></h4><p>Initially I joined a new team which was still being formed, where the team’s vision was undecided, but goals were clearly defined. My teammates had a mix of experience and we worked on building some tracing and logging systems.</p><p>After a few weeks of working here, I was asked to work with another team where only two people were working on a project and they were lacking bandwidth for the new requirements. The tech in the company was still evolving, and the team was fairly new. They just had the initial version of the system deployed and serving.</p><p>Those two folks were fully on maintenance mode for the existing system and couldn’t accept any more new requirements. A few other senior folks and I came to the rescue. We immediately started working on new requirements, revamping existing systems and building new features. It was a really fun beginning.</p><p>I have been working with the same team ever since — a two people team that has now grown out to be a big team of 16+ engineers with great impact.</p><p>The growth of the team, system and process is really astonishing. Now that I sit and write this, thinking about the things my team has achieved really surprises me. We have experimented with different ways of running sprint, testing, deployment, automation and stand-ups.</p><p>From manually deploying services in VM’s to having weekly deployment, automated CI pipelines, test, and kubernetes solutions with our own cluster managed by the team itself.</p><p>Currently we are running a two week sprint, a standup everyday where we each keep a personal ledger of tasks where we update the previous day’s activities date, we talk about what our tasks for the day will be, discuss on any blockers and clarifications and make announcements. Can’t leave out Jira when talking about sprint and tasks, right? So I’ll say it — I don’t like it as of now. 😂</p><p>I work out of Bangalore office and the team is in Singapore. We don’t just work, it’s also about having fun virtually from our homes and finding the right balance.</p><h4><strong>What did I learn?</strong></h4><blockquote>TL;DR: I learnt many things, but importantly to test the damn thing well.</blockquote><p>Building a system at such a scale is easy if you know how to do it right, but that’s the beauty of our industry: We don’t know how to do it right. 🤷‍♂️</p><p>We follow certain processes with certain expectations, we iterate them until we get something stable working for that moment until the next unforeseen thing comes up.</p><p>Question we ask is:</p><blockquote>How do you know what is right or what you have done is right? You test it. Period.</blockquote><p>Gojek follows test driven development; the entire org depends on it. Each code the developer writes has to be unit tested, then functional, integration, and a whole other bunch of testing processes. That’s not enough, there is a QA doing extensive testing on the feature you are developing. There are automated test suites running in the CI looking for issues.</p><p>With all these things, the guy who hated the test 4 years ago, now loves it. I started my career where building projects would have `-<em>DskipTests</em>` flag set , you just get things done without testing, if it runs, great. Ship it. I have a project which was built without proper testing and it’s been a nightmare for me to refactor, improve, or address issues. Procrastinating over it as of now for 3 years.</p><p>I started writing tests after I joined Gojek, which changed the way I was writing programs. As a developer we (I?) tend to write complex stuff, shove all the things we think need, make it complex, thinking that hard-to-understand programs would make one a great programmer.</p><blockquote>I thought that’s good programmer skills — writing programs in complicated ways. Well, I was wrong.</blockquote><p>Writing simple, readable, testable code is what good programmer skills are.</p><p>Unit testing each code you write enables you to do the same thing, your thinking changes, you will end up deleting codes that aren’t important because while testing you realise you don’t need to support that at all or you are too lazy to cover the case in a test which nobody will use (YAGNI). You will start writing simple, concise functions which do one job and do it well. Forces you to refactor. All in all, you would have a well tested unit.</p><p>This reduces the risk of issues going to production and blowing it up or launching nuclear warheads. (may be still possible, but reduces the risk significantly).</p><p>I used to say that or believed that “if the system doesn’t have enough issues and you haven’t debugged it, then you don’t understand the system at all”, because this is how I have learnt many systems in previous organisation, but that changed here.</p><p>Writing tests eliminates future risk/bugs, but it also helps to find the existing one too. You don’t have to sit and step through each line of code to see where shit hit the fan or start writing printfs.</p><p>Now you may ask, what guarantee is that you have written your test right? What guarantee is that your test case is actually covering edge cases?</p><p>That’s why we have reviewers or a pair, every change to the system is reviewed by the peer, one who has an understanding of the system and the feature you are developing. He is also responsible as you are for the development and deployment of the feature.</p><p>Another part of the development process in Gojek is that it also advocates for pair programming. Two developers work/write codes, help bounce ideas around and reduce bus factor. Though the current situation of WFH made it a bit hard for us to follow it, that’s something which I have done in the beginning.</p><p>There are lot more source of learning in Gojek, not just on the work you do. We have dedicated channels where people share their findings, RCA session where teams go about the issue they face, fixes, learnings with different teams.</p><p>So far I have talked about what I have learned while working at Gojek, but here’s…</p><h4><strong>What Gojek did for me</strong></h4><p>Well to begin with, it pays my bills and gives amazing perks, duh!</p><p>With all the work, I have ample time at hand, and I spend a lot of it on learning and building systems on my own. I have developed quite a few personal projects in my free time, and experimented with many tools, languages.</p><p>The whole team and company respects every individual’s time, they care for the employee.</p><p>The team and the leaders care about their teammates. Micromanagement is unheard of, instead they ask how they can help finish a task or get rid of blockers. The one-on-one meetings which team leads hold with teammates is really amazing, with fruitful discussion, sorting out issues, etc.</p><p>After working for years, I started to feel bored with what I was doing, and felt that it wasn’t enough. So, I started crafting my own projects, built systems, experimented with any tools. Trying to code more backfired and I started inching towards a burnout. I was looking for solutions.</p><p>Upon reaching out to both my current and previous team leads, I was advised well and they assured it’s not too uncommon and I needed to take time and try things not related to computer science.</p><p>So yep, when they say “Hard to get in, harder to leave”, I know it’s true. Seeing the progress on how a company has an impact on individuals in Southeast Asia really amazes me and makes me proud to be part of the company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8zvrbKMwNTIObVAFlO0D6A.png" /></figure><h4>About the author</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*qwBljkIqI-HKuZ2cUbOFWw.jpeg" /></figure><p>Nishanth Shetty is our GoTroop who builds pricing systems at Gojek. You can find him on LinkedIn, <a href="https://www.linkedin.com/in/nishanthspshetty">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPWwoozDxeKoXMtXEyrtLw.png" /></figure><p>Read more stories from our vault, <a href="https://www.gojek.io/blog">here</a>.</p><p>We’re hiring. Always. 🖖<br>Click below to view open job positions.</p><figure><a href="https://bit.ly/gojekjobsblogs2022"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*c2_eIvR2WDUWnP6YmzcGpQ.jpeg" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ccdff6730c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gojekengineering/two-years-and-a-thousand-learnings-later-3ccdff6730c9">Two Years And A Thousand Learnings Later</a> was originally published in <a href="https://medium.com/gojekengineering">Gojek Product + Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Drafting a JVM With Rust]]></title>
            <link>https://medium.com/gojekengineering/drafting-a-jvm-with-rust-1de3129997c9?source=rss-f5de13ae96b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1de3129997c9</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[virtual-machine]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[rustlang]]></category>
            <dc:creator><![CDATA[Nishanth Shetty (@nsp)]]></dc:creator>
            <pubDate>Tue, 04 Feb 2020 04:07:09 GMT</pubDate>
            <atom:updated>2020-02-04T04:07:09.919Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to implement a stack-based virtual machine.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mp9IMlrMIiqh3Y0TXQqt9Q.jpeg" /><figcaption><a href="http://bit.ly/2vFxQDV">gojek.jobs</a></figcaption></figure><p>Virtual machines emulate computer systems by providing a high-level abstraction of the underlying operating system, and therefore provide the same functionality.</p><p>Broadly, they’re of two types — one is the <strong>System Virtual Machine </strong>that emulates an<strong> </strong>entire system and allows users to run an Operating System (ex: VMWare).</p><p>The other kind is the <strong>Process Virtual Machine, </strong>or <em>Managed Runtime Environment</em> (MRE), which runs a user program on a host machine. Ex : JVM, CLR (Common Language Runtime) a .Net runtime.</p><p>We’ll be focusing on the latter in this post.</p><h3>Implementation</h3><p>There are two ways to implement the Virtual Machine — Stack-based and Register-based. JVM is a stack-based virtual machine whereas Dalvik VM is register-based.</p><p>Stack-based VM operates on two well-known stack operations — push and pop. Data is pushed on to the stack and operation is performed on the data at the top of the stack. The result will be pushed back to stack.</p><p>We will try to implement our own stack-based virtual machine which will perform basic arithmetic operations. But first, we need to define runtime specification for our VM.</p><blockquote>1. VM Will perform addition, subtraction, multiplication, and division.</blockquote><blockquote>2. VM will use wordcode of size 32bit for data and instruction set. Bytecode for the VM will use 32bit word size, hence the wordcode.</blockquote><blockquote>3. VM will reach halt state when `HALT` instruction is encountered and every program must end with `HALT` instruction.</blockquote><h3>Operation</h3><p>VM Performs fetch-decode-execute cycle,</p><ul><li>fetch Retrieve data/instruction from VM program memory</li><li>decode Identify the type of the data fetched, either VM Instruction(Operation Instruction) or Data</li><li>execute Perform the operation based on the decode type.</li></ul><p>Operation Instruction Performs the operation defined by the instruction on the data present in the top of the stack.</p><p>Data Push the data to stack.</p><h4>Defining Instruction Set</h4><p>As per specification, our VM will use 32 bit to store the instruction and data. To separate instruction and data in our program, we need to have a way of communicating the same to the VM. We will allocate 2 bits to indicate the type; that leaves us with 30bits to store the data (i.e we can have numbers value range 2³⁰ in positive and negative values).</p><p>We will use 2 MSB bits for code type as follows:</p><pre>|bits| Type |<br>|----|-----------------|<br>| 00 | Data (Positive number)|<br>| 01 | Operation Instruction (OI)|<br>| 11 | Data (negative number)|<br>| 10 | undefined|</pre><p>Stack-VM Instructions:</p><pre>0x40000001    ADD           <br>0x40000010    SUB           <br>0x40000011    MUL           <br>0x40000100    DIV           <br>0x40000000    HALT</pre><p>The code to perform 22 + 32 will look like this:</p><pre>| Hex word  | VM Value |<br>|0x00000020 | Data (32 in decimal)|<br>|0x00000016 | Data (22 in decimal)|<br>|0x40000001 | OI (ADD)|<br>|0x40000000 | OI (HALT)|</pre><p>Now that we have defined basic specification for the VM, its time for some code. We will use <a href="https://www.rust-lang.org/">rust-lang</a> for the implementation.</p><blockquote>Create new Rust project with cargo new Stack-VM</blockquote><p>Before we start, we will make following assumptions:</p><ol><li>VM uses little endian byte order.</li><li>VM uses separate memory for runtime stack and program memory to load program.</li></ol><p>Add the byteorder crate to your cargo.toml file as follows:</p><pre>[dependencies]<br>byteorder = &quot;1.0.0&quot;</pre><p><strong>Create runtime stack</strong></p><p>Create lib module in your src folder, we will create all required modules here.</p><p>Create stack module and implement the stack with its primitive operation — push() and pop() .</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71ca1871a848c0e7549f48ddd4313c6b/href">https://medium.com/media/71ca1871a848c0e7549f48ddd4313c6b/href</a></iframe><p><strong>Create VM Runtime.</strong></p><p>Stack VM will have its internal state structure to store current program counter, vm state, runtime stack, and program memory. Create the module called vm and create the below struct:</p><pre>pub struct VM {<br>   <br>    //program counter<br>    pc: usize,<br>   <br>    //typ of value read<br>    type_info: i32,<br><br>    //data/instruction<br>    data: i32,<br>    running: bool,<br>    stack: VMStack,<br>    program_memory: Vec&lt;i32&gt;,<br>}</pre><p>Initialise the VM with default state as follows:</p><pre>VM {<br>    program_memory: Vec::<em>new</em>(),<br>    pc: 0,<br>    type_info: 0,<br>    data: 0,<br>    running: true,<br>}</pre><p>We will implement a few methods on VM as follows:</p><h4>1. load_program</h4><p>Loads the user program from file to VM program_memory . When we run fetch as a first task or run cycle, it will advance the program counter, so PC will start the code execution at memory index 1. Hence, we need to load our program to code area starting from index 1. For this reason, we will push magic bit 0xBADC0DE to first position of vector, at index 0.</p><pre>pub fn load_program(&amp;mut self, instructions: &amp;Vec&lt;u32&gt;) {<br>    debug!(&quot;Loading program...&quot;);<br>    //Insert magic bits to beginning of the program memory, so that we can start pc from 1<br>    self.program_memory.push(0xBADC0DE);<br>    for instruction in instructions {<br>        self.program_memory.push(*instruction);<br>    }<br>    debug!(&quot;Memory content : {:?}&quot;, self.program_memory);<br>}</pre><h4>2. fetch</h4><p>Loads instructions from the program memory. We will just advance the program counter in this step:</p><pre>fn fetch(&amp;mut self) {<br>   if self.pc &lt; self.program_memory.len() {<br>        self.pc += 1;<br>    } else {<br>        panic!(&quot;Incomplete code execution, memory boundary reached   without reading HALT instruction&quot;);<br>    }<br>}</pre><h4>3. decode</h4><p>Extracts the operation instruction and data from the word(byte):</p><pre>fn decode(&amp;mut self) {<br>    let word = self.program_memory[self.pc];<br>    self.data = VM::<em>get_data</em>(word);<br>    self.type_info = VM::<em>get_type</em>(word);<br>}</pre><h4>4. run</h4><p>Executes fetch-decode-execute cycle till program halts.</p><h4>5. exec</h4><p>Based on the decode information, it either pushes the data to stack or performs operation if the code type is Operation Instruction.</p><pre>fn exec(&amp;mut self) {<br>    if self.current_instruction_type() == <em>CODE_TYPE_DATA_POSITIVE </em>|| self.current_instruction_type() == <em>CODE_TYPE_DATA_NEGATIVE </em>{<br>        debug!(&quot;Instruction type Data ({} = {} ) &quot;, self.current_instruction_type(), self.current_data());<br>        self.stack.push(self.data);<br>    } else {<br>        //execute instruction<br>        debug!(&quot;Instruction type Operation ({}) , &quot;, self.current_instruction_type());<br>        self.do_primitive();<br>    }<br>}</pre><h4>6. do_primitive</h4><p>Executes instruction (do_primitive()).</p><p>This method will perform the arithmetic operation as defined in the spec. It will pop values from the top of the stack, perform the operation specified in instruction, and push the result back to the top of the stack. This is the Arithmetic Unit of our VM.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e51b94672e996e2e43a185c95e601fd/href">https://medium.com/media/9e51b94672e996e2e43a185c95e601fd/href</a></iframe><p>Put together all the above, and create main.rs as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14abfdfc72d170dd8d1ebb6574552fcc/href">https://medium.com/media/14abfdfc72d170dd8d1ebb6574552fcc/href</a></iframe><p>To run, we need to generate binary for our VM. We can use <a href="https://github.com/NishanthSpShetty/stack-langc">stack-langc</a> compiler to generate the binary for us.</p><p>stack-langc uses simple postfix expression to generate the binary, so in order to get our program to add two numbers, create program.vm file with below content:</p><pre>32 22 +</pre><p>To generate binary, move to stack-langc repo and run the following:</p><pre>$cargo run program.vm</pre><p>This will generate a.out binary file with Stack-VM bytecode.</p><p>$hexdump a.out will show the following content for the a.out.</p><pre>0000000 20 00 00 00 16 00 00 00 01 00 00 40 00 00 00 40<br>0000010</pre><p>Now run Stack-VM program using:</p><pre>$cargo run a.out</pre><p>As we are using debug mode, you should see debug logs printed with stack content and the operation the VM is performing.</p><pre>Memory content : [195936478, 32, 22, 1073741825, 1073741824]<br>Executing instructions...<br>Instruction type Data (0 = 32 ) <br>Instruction type Data (0 = 22 ) <br>Instruction type Operation (1) , <br>[ ADD ] : 22 32 <br>TOS now : 54<br>Instruction type Operation (1) , <br>[ HALT ] : Stopping VM</pre><p>That’s all there is to it. This exercise is an attempt to understand Rust and stack-based VM implementation better. You can find the full project <a href="https://github.com/NishanthSpShetty/Stack-VM/"><em>here</em></a> at my Github page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yigf1nGxRKjzV7vIC_YkYg.png" /></figure><p>Like what you read? <a href="https://mailchi.mp/go-jek/gojek-tech-newsletter">Sign up for our newsletter</a> and get our stories in your inbox every week. 🖖</p><figure><a href="http://bit.ly/2vFxQDV"><img alt="Job Openings at Gojek" src="https://cdn-images-1.medium.com/max/803/1*XSAkmYAomyHp6_18rjjAfA.jpeg" /></a><figcaption>gojek.jobs</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1de3129997c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gojekengineering/drafting-a-jvm-with-rust-1de3129997c9">Drafting a JVM With Rust</a> was originally published in <a href="https://medium.com/gojekengineering">Gojek Product + Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A TOAST from PostgreSQL]]></title>
            <link>https://medium.com/gojekengineering/a-toast-from-postgresql-83b83d0d0683?source=rss-f5de13ae96b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/83b83d0d0683</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Nishanth Shetty (@nsp)]]></dc:creator>
            <pubDate>Tue, 21 Jan 2020 04:20:57 GMT</pubDate>
            <atom:updated>2020-01-21T04:20:57.195Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Postgres handles data storage of oversized attributes.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_WsomB3w68tF-b4cFUyPQg.jpeg" /><figcaption><a href="http://bit.ly/2RAFmHz">gojek.jobs</a></figcaption></figure><p>PostgreSQL is a powerful open source object relational database system, which has been widely adopted as a primary option for many applications.</p><p>In this post, we’ll take a look at how Postgres handles database storage and caching.</p><h3>First, the basics</h3><p>PostgreSQL database loads all data into page cache, which is different from OS Paging. A page cache of size 8KB is used for storing tuples, indexes, and Query Execution plans. Even the WAL are written to 8KB pages.</p><p>While the underlying stack may use a different block size, typically Linux (and most Operating Systems) use 4KB page size. So when PostgreSQL writes data to disk, the underlying stack may break into a smaller chunk depending on its block size (it may be 2 blocks of 4KB or 16 blocks of 512MB). But this is in the case of Full Page Write (FPW)</p><p>PostgreSQL limits the tuple (or row) to be stored within this page, which seems like a limitation seeing the size of the data users may want to store and is allowed by the database. But this limitation doesn’t enforce the size restriction on database rows. PostgreSQL handles this differently when the data spills out of the boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*qgACX776CUx213se.png" /></figure><p>PostgreSQL allows data columns of larger size with types like <em>varchar</em>, <em>text</em>, <em>bytea</em>, <em>json</em>. Let’s understand how it manages to put large data in a fixed size page cache.</p><h3><strong>TOAST — </strong>The Oversized-Attribute Storage Techniques</h3><p>Toast is a mechanism in PostgreSQL to handle large chunks of data to fit in page buffer. When the data exceeds <em>TOAST_TUPLE_THRESHOLD</em> (2KB default), Postgres will compress the data, trying to fit in 2KB buffer size. If the compressing of the large column data does not lead to smaller block (&lt;2KB), it will be split into smaller chunks.</p><p>This is enabled by default and all tables will have the toast table associated with it. You can check the associated toast table in pg_class using the following query:</p><pre>select relname from pg_class where oid = (select reltoastrelid from pg_class where relname=’TABLE_NAME’)</pre><p>The table name will be pg_toast_$(OID) where oid is the toast table oid, reltoastrelid of original table.</p><p>The toast table is in pg_toast schema, so to query you need to use:</p><pre>select * from pg_toast.pg_toast_$(OID)</pre><p>The above query will print the toasted chunk_id, chunk_seq, chunk_data, if present.</p><blockquote><strong><em>Note</em></strong><em>: </em><em>pg_toast_* table entry is only for data which is split into multiple chunks.</em></blockquote><h4><strong>Out of line storage</strong></h4><p>If data does not fit in a single page cache even after compression, then the out of line storage technique is used, splitting data into multiple chunks of smaller size.</p><p><strong>Toast table storage strategy</strong></p><p>All database columns have the storage strategy associated with them, explained as follows:</p><blockquote><strong><em>1.</em></strong><em> </em><strong><em>Plain</em></strong></blockquote><blockquote><em>Does not allow out of line storage or compression, ex : numbers, char type.</em></blockquote><blockquote><strong><em>2.</em></strong><em> </em><strong><em>Extended</em></strong></blockquote><blockquote><em>Allows out of line storage and compression, default for most data types that can use toast. Compression is attempted first and then if that does not help to fit the data in page, out of line storage is used.</em></blockquote><blockquote><strong><em>3. External</em></strong></blockquote><blockquote><em>Allows out of line storage, not compression. Makes substring operation faster on text and bytes columns. Has storage space penalty.</em></blockquote><blockquote><strong><em>4. Main</em></strong></blockquote><blockquote><em>Same as external but out of line is performed when there is no way to make the column small enough to fit in page.</em></blockquote><p>All the columns will not take part in toast, as the implicit size limit of types such as <em>char</em>, <em>numbers</em> will not hold large data.</p><p>The types which have variable length representation — such as <em>text</em>, <em>varchar —</em> are ideal toast candidates.</p><p>This will ensure the data can be loaded into single page buffer. While this is useful when storing large amounts of data, it does add some performance overhead of compression-decompression.</p><blockquote>However, this same overhead can be used to our advantage.</blockquote><p>Let’s say we have a remote data center storing the data. If we query the toasted data over the network, the data size to be shipped is smaller and decompression time can be less, compared to network latency for large raw data.</p><p>Hopefully, this gave you an understanding of how Postgres handles storage and caching. So let me leave you with a question:</p><blockquote>Q: Why are the attributes of pg_toast_* storage strategy plain?</blockquote><p>Leave your answers in the comments, and let me know if this was helpful. If you’d like to know more about this topic, check out the links below:</p><blockquote>1. <a href="https://www.postgresql.org/docs/current/storage-toast.html">https://www.postgresql.org/docs/current/storage-toast.html</a></blockquote><blockquote>2. <a href="https://blog.codinghorror.com/maybe-normalizing-isnt-normal/">https://blog.codinghorror.com/maybe-normalizing-isnt-normal/</a></blockquote><blockquote>3. <a href="http://www.25hoursaday.com/weblog/CommentView.aspx?guid=cc0e740c-a828-4b9d-b244-4ee96e2fad4b">http://www.25hoursaday.com/weblog/CommentView.aspx?guid=cc0e740c-a828-4b9d-b244-4ee96e2fad4b</a></blockquote><blockquote>4. <a href="https://www.citusdata.com/blog/2013/04/30/zfs-compression/">https://www.citusdata.com/blog/2013/04/30/zfs-compression/</a></blockquote><blockquote>5. <a href="https://madusudanan.com/blog/understanding-postgres-caching-in-depth/">https://madusudanan.com/blog/understanding-postgres-caching-in-depth/</a></blockquote><blockquote>6. <a href="https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes/">https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes/</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yigf1nGxRKjzV7vIC_YkYg.png" /></figure><p>Liked this story? if you’d like our blog posts delivered straight to your inbox, <a href="https://mailchi.mp/go-jek/gojek-tech-newsletter">sign up for our newsletter</a>!</p><figure><a href="http://bit.ly/2RAFmHz"><img alt="Gojek Job Openings" src="https://cdn-images-1.medium.com/max/803/1*XSAkmYAomyHp6_18rjjAfA.jpeg" /></a><figcaption>gojek.jobs</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83b83d0d0683" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gojekengineering/a-toast-from-postgresql-83b83d0d0683">A TOAST from PostgreSQL</a> was originally published in <a href="https://medium.com/gojekengineering">Gojek Product + Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A TOAST from PostgreSQL]]></title>
            <link>https://medium.com/@nishanthspshetty/a-toast-from-postgresql-f102014ad279?source=rss-f5de13ae96b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f102014ad279</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[toast]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Nishanth Shetty (@nsp)]]></dc:creator>
            <pubDate>Mon, 30 Dec 2019 05:53:54 GMT</pubDate>
            <atom:updated>2019-12-30T05:53:54.536Z</atom:updated>
            <content:encoded><![CDATA[<p>PostgreSQL is powerful open source Object relational database system, PostgreSQL got a wide adoption as a primary option for most of the applications.</p><h4>A look at database storage and caching.</h4><p>PostgreSQL database loads all data into page cache, which is different from the OS Paging. A page cache of size 8KB is used for storing tuples, indexes and Query Execution plans. Even the WAL are written to 8KB pages.</p><p>While the underlying stack may use different block size, typically Linux uses 4KB page size(most of all the OS). So when the PostgreSQL writes data to disk, underlying stack may break into smaller chuck depending on its block size, it may be 2 blocks of 4KB or 16 blocks of 512MB. But this is the case of full page write (FPW)</p><p>PostgreSQL limits the tuple(/row) to be stored within this page, which seems like a limitation seeing the size of the data users may want to store and allowed by the database. But this limitation doesn’t enforce the size restriction on database rows, Postgres handles this differently when the data spills out of the boundary.</p><figure><img alt="PostgreSQL Page Cache Structure" src="https://cdn-images-1.medium.com/max/724/1*sTM0ZGsvr4lRLJ1Qa3uWsQ.png" /></figure><p>PostgreSQL allows columns data of larger size with types like <em>varchar</em>, <em>text</em>, <em>bytea</em>, <em>json</em>. Lets understand how Postgtres manages to put large data in such fixed size page cache.</p><p><strong>TOAST</strong></p><p><em>The Oversized-Attribute Storage Techniques</em></p><p>Toast is a mechanism in the PostgreSQL to handle large chunk of data to fit in page buffer. When the data exceeds <em>TOAST_TUPLE_THRESHOLD</em> (2KB default), Postgrtes will compress the data trying to fit in 2KB buffer size, if the compressing of the large column data is not leading to smaller block(&lt;2KB) it will be split into smaller chunks.</p><p>This is enabled by default and all the table will have the toast table associated with it. You can check the associated toast table in <strong><em>pg_class</em></strong> using the following query.</p><pre>select relname from pg_class where oid = (select reltoastrelid from pg_class where relname=’TABLE_NAME’)</pre><p>the table name will be <strong><em>pg_toast_$(OD)</em></strong> where <em>oid</em> is the toast table <em>oid</em>, <em>reltoastrelid</em> of original table.</p><p>The toast table is in pg_toast schema, so to query you need to use</p><pre>select * from pg_toast.pg_toast_$(OID)</pre><p>The above query will print the toasted chunk_id, chunk_seq, chunk_data if present.</p><blockquote><strong>Note</strong>: pg_toast_* table entry is for only those data which are split into multiple chunks.</blockquote><p><strong>Out of line storage</strong></p><p>data which does not fit in a single page cache even after the compressed, then the out of the line storage technique is used, splitting data into multiple chunks of smaller size.</p><p>Toast table storage strategy</p><p>All database columns has the storage strategy associated with it, explained as follows</p><blockquote>1. <strong>Plain</strong></blockquote><blockquote>Does not allow out of line storage or compression, ex : numbers, char type.</blockquote><blockquote>2. <strong>Extended</strong></blockquote><blockquote>Allows out of line storage and compression, default for most data types that can use toast. Compression is attempted first and then if that dint help to fit the data in page, out of line storage is used.</blockquote><blockquote>3. <strong>External</strong></blockquote><blockquote>Allows out of line storage, not compression . Makes substring operation faster on text and bytes columns. With storage space penalty.</blockquote><blockquote>4. <strong>Main</strong></blockquote><blockquote>Same as external but out of line performed when there is no way to make column small enough to fit in page.</blockquote><p>All the columns will not take part in toast, as implicit size limit of types such as char, numbers which will not hold large data.</p><p>The type which has variable length representation such as <em>text</em>, <em>varchar</em> is the toast candidate.</p><p>This will make sure that the data can be loaded into single page buffer, this is useful when storing large data, it may add overhead of compression-decompression but the same overhead can be used for our advantage, how ?</p><p>Lets say we have remote data center storing the data and if we query the toasted data over the network, the data size to be shipped is smaller and decompression time can be smaller compared to network latency for large raw data.</p><blockquote>Q: why is the storage strategy of pg_toast_* attributes are plain?</blockquote><p>Follow up on these links to know more</p><p>1. <a href="https://www.postgresql.org/docs/current/storage-toast.html">https://www.postgresql.org/docs/current/storage-toast.html</a></p><p>1. <a href="https://blog.codinghorror.com/maybe-normalizing-isnt-normal/">https://blog.codinghorror.com/maybe-normalizing-isnt-normal/</a></p><p>2. <a href="http://www.25hoursaday.com/weblog/CommentView.aspx?guid=cc0e740c-a828-4b9d-b244-4ee96e2fad4b">http://www.25hoursaday.com/weblog/CommentView.aspx?guid=cc0e740c-a828-4b9d-b244-4ee96e2fad4b</a></p><p>3. <a href="https://www.citusdata.com/blog/2013/04/30/zfs-compression/">https://www.citusdata.com/blog/2013/04/30/zfs-compression/</a></p><p>4. <a href="https://madusudanan.com/blog/understanding-postgres-caching-in-depth/">https://madusudanan.com/blog/understanding-postgres-caching-in-depth/</a></p><p>6. <a href="https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes/">https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f102014ad279" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>