<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Krishna on Medium]]></title>
        <description><![CDATA[Stories by Krishna on Medium]]></description>
        <link>https://medium.com/@krishna.b.chy?source=rss-ec718819a032------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5XZI60oo7sHX46x3WUzKRw.png</url>
            <title>Stories by Krishna on Medium</title>
            <link>https://medium.com/@krishna.b.chy?source=rss-ec718819a032------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:21:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@krishna.b.chy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Big Bang theory of production release]]></title>
            <link>https://medium.com/@krishna.b.chy/the-big-bang-theory-of-production-release-75a38c7b4ab?source=rss-ec718819a032------2</link>
            <guid isPermaLink="false">https://medium.com/p/75a38c7b4ab</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-release]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <dc:creator><![CDATA[Krishna]]></dc:creator>
            <pubDate>Wed, 16 Nov 2022 11:21:44 GMT</pubDate>
            <atom:updated>2022-11-16T14:46:54.936Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1jTMnN5ZdKMFOQG0" /><figcaption>Photo by <a href="https://unsplash.com/@mischievous_penguins?utm_source=medium&amp;utm_medium=referral">Casey Horner</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It is not necessary that – what has worked for others or what has been written in books or technical blogs for that matter, might work for you too. While they may cater to a wide variety of problems, you may need an indigenous approach or an approach that may sound revolutionary or anti- pattern to the world at that point of time. But that’s OK. Nobody knows your problems better than you, no one knows all the technical details of your system, impacts and implications of the change, better than you. The timeline constraints, the work force constraints, the urgency and <strong>everything else that impact the change and that are impacted by the change</strong> – you know them the best.</p><p>Recently, the Engineering team at <a href="https://eltropy.com/">Eltropy</a> did the biggest production release ever in the history of the organization in terms of technology. It had been a long awaited technical debt. <a href="https://grails.org/"><strong>Grails</strong></a> which had been the back bone of the back-end system was bid farewell for certain production line. Grails is one of the most wonderful web application frameworks out there and it helped us to build, ship and iterate quickly. Thereby it contributed immensely to Eltropy’s growth, no doubt. Big shout out to Grails.</p><h4>But then, for every sunrise there ought to be a sunset.</h4><p>Primary reason for the decision was that — the <strong>version</strong> of Grails we had been using was approaching its ‘end of life’ and upgrade to the latest version needed significant effort. At the same time, ‘<a href="https://go.dev/">Golang</a>’ has been our ‘go to’ language for any new implementation unless there is specific reason or requirement. ‘Go’ has served us very well in every aspect.</p><p>We had two options in terms of release strategy.</p><ol><li>We could do release in phases where we could move things to new tech stack across multiple releases.</li><li>We could go the Big Bang route.</li></ol><p>We needed to ship it fast. There were many other technical debts and infrastructural improvements pending to this activity. First strategy, as sensible and attractive as it may sound, was probably not meant for us this time for the simple reason that it would need more time, energy and engineering for building, testing and shipping but unfortunately these were luxuries we wished we had at the moment with so many business requirements lined up. Normally we release big features in phases.</p><p>After multiple brain storming sessions we decided to go for a big bang approach provided we had an option to roll things back should something go wrong in worst case and that too - only if a ‘Fast Fix Forward’ was not an option at all.</p><p>We were not going along the general recommendation of releasing in phased manner this time. This was not an easy decision. But then challenging times demand tough decisions, isn’t it?</p><p>We knew the complexity of our system, impacts of the changes, the edge cases and almost everything around it and hence we knew what we were up-to. One may call it ‘a well calculated risk’, but we believed it was a ‘no risk’ release since we had almost every control in our hands to undo and re-do. Hence the confidence.</p><p>It needs knowledge of the system, courage, conviction and data driven impact analysis to take such bold decisions. Our CTO <a href="https://www.linkedin.com/in/rakesh-goyal-21946422/">Rakesh Goyal</a>, known for taking bold technical decisions had all of these thought through. We discussed every possibilities, the pros and cons. Everyone rallied behind him to make this happen.</p><p>It was a mega project, ‘Grails migration’ as we called it, that went through active development for many months. We had few fantastic engineers set the ground and complete most of the development work. Most of them were new to the tools and technologies we use at Eltropy. Kudos to them for picking it up real quick and starting contributing significantly. Our CTO is known for empowering developers by giving them complete autonomy to drive the show. This project was no exception. Unfortunately some of the main contributors had to leave in the mid of the project and that left us in little tricky situation. But again with everyone’s support and management’s flexibility in terms of timeline, we pulled it off.</p><p>Few <strong>random</strong> points from my experience that others may benefit from -</p><p>API preservation — We made sure we did not change route and contract of any existing APIs. Unused and obsolete APIs were removed. Any unhappy change required due to conflicting route with existing route name-spacing were handled at the load-balancer level which is HAProxy in our case. This made sure that front end did not need any change as far as consuming APIs was concerned.</p><p>Pre-validation of data migration — We also had some database migrations that were required to be run as part of the the new RBAC feature that we were shipping as part of this exercise. The Q&amp;E team made sure that this was run in a copy of the production database in order to avoid any runtime issues during migration in production database. Any issue found because of data inconsistency or something else were rectified accordingly at this stage itself.</p><p>Build to rollback if required — The critical decision that was taken from day one was that, we would port business logic ‘as-it-is’ to the new tech stack. No optimization, no performance improvement as part of this exercise. No change in the underlying logic of APIs or services. This made sure that we did not have to make any destructive (update, delete etc) changes to the database as part of the exercise. Hence if at all, we had to roll back the deployment for some unpredictable reason, we could just roll back the code and we could very well be in a state as good as we were before. The only change with respect to database was the new set of migrations that were needed to be run in order to create schema and populate seed data for the RBAC feature that we were implementing as part of this exercise. Since these were independent entities, they could be safely undone if required.</p><p>Centralized deployment checklist — The deployment activity had an array of tasks to be performed on the D-day unlike usual releases. Many of them could have been automated and that would have saved us many hours thereby giving us more time to test post deployment. For some reason, we could not automate those tasks. But the good thing was, the whole exercise was performed by our Q&amp;E team in independent environments multiple times during testing phase by following the checklist as bible. Most of the possible error cases were already handled and hence we did not see any surprise during deployment despite of innumerable tasks. But this was something we should have done better by automating them and definitely a lesson for tomorrow.</p><p>Project Management — One major challenge was to accommodate new changes coming in as part of addition of new features or enhancements to the product to this parallelly running ‘Grails migration’ project. The project manager made sure that newly added code were also ported to this new code base. We had not frozen the Grails code base yet as it was still running in production and we had to provide support for any critical requirements coming in. The project manager along with other stake holders made sure that only critical changes were allowed in and rest were queued up to be done post this release based on priority. Tussle between product and engineering team is not a new phenomenon in a software company but wise teams do reach certain agreement that suits best to both the sides. There is no replacement for an understanding team.</p><p>Of course, daily stand ups, regular communication with every stake holders, rigorous testing, integrated testing in fresh new environments etc were always part of the process.</p><p>We had all the support needed from the management and other teams for this epic execution. Everybody rallied around us and we can not thank them enough.</p><p>Did it go all good ? I wouldn’t say so. Post deployment, we did have couple of countable issues. But they were very minor ones in terms of fix. Once discovered and reproduced, they were fixed, tested and deployed then and there. This was part of the strategy. But we did not see any major issues so far. This is where test automation with full coverage plays a big role. Despite of it being a monstrous release, it was a greatly successful release for us.</p><p>Now that the biggest technical debt is out of our way, we are better equipped and in better position to build all those things that we had been wanting to build for a long time and could not because of these legacy debts.</p><p>I guess this is the story of most of the start ups where you start small then at some point, you have to start building it for scale as your business scales. Isn’t it?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75a38c7b4ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beautiful nightmares — Technical investigation of production issues]]></title>
            <link>https://medium.com/@krishna.b.chy/beautiful-nightmares-technical-investigation-of-production-issues-f644ae91d98?source=rss-ec718819a032------2</link>
            <guid isPermaLink="false">https://medium.com/p/f644ae91d98</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Krishna]]></dc:creator>
            <pubDate>Wed, 27 Apr 2022 16:30:39 GMT</pubDate>
            <atom:updated>2022-04-28T06:38:07.261Z</atom:updated>
            <content:encoded><![CDATA[<h3>Technical investigation of production issues in software — Beautiful nightmares ?</h3><p>Technical investigation of software production issues, sometimes, gives me a temporary pleasure of role playing ‘<strong>Sherlock Holmes</strong>’. Or at least that’s what I would like to believe for the time being while I am on it, despite of it being one of the most <strong>undervalued tasks </strong>in software development. While some of them can be easy and straightforward, most of them can turn out to be nightmares. It will not be an exaggeration to say that technical investigation is one of the most intimidating and daunting task in software ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I_W8AaJHzt00XqRP" /><figcaption>Photo by <a href="https://unsplash.com/@lunarts?utm_source=medium&amp;utm_medium=referral">Volodymyr Hryshchenko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Effective treatment depends solely on the right diagnosis. Isn’t it? That’s precisely what we do when we do an investigation in order to find the root cause of an issue. Finding the root cause of an issue is always the first and right step towards its resolution. On the other hand, if we are not able to find the root cause, we may end up fixing the symptoms and the issue can resurface anytime in future, may be with a bang, who knows?</p><p><em>Via this medium, I would like to share my experience and some simple strategies that has helped me over the years in some of the toughest and most difficult investigations in our production environment. Thankfully I have been able to find the root cause of 100 percent of the issues that I have ever got myself into at </em><a href="https://eltropy.com/"><em>Eltropy</em></a><em>.</em></p><p>If every day is a happy day, we would never have to dig the dirt. But then life is full of uncertainties, isn’t it? There may be unhappy situations where there is no other option but to indulge into deep investigation to figure out the culprit so that a right decision can be taken in order to fix the issue or resolve the crisis. It could be some hideous bug that brought the servers to its knees, an abnormal load on database, a sudden drop in performance that affected the valued customers during their business hours or signs of malicious attempts to gain access to critical resources. In order to handle them effectively we need to know the root cause of the issue which, if not handled on time, can come back and hit hard when we least expect them.</p><blockquote><em>While we mostly choose to share adoption of latest technology to the infrastructure, complex features built and shipped to productions, fancy new interfaces and what not, today I choose to select ‘Technical Investigation’ as the topic to show solidarity and appreciation to fellow engineers who do this dauntingly complex task behind the scenes that saves the ship from sinking on many occasions but still continues to be an </em><strong><em>undervalued</em></strong><em> effort in most of the software companies. Most of the times if not always, it does not earn one much of an appreciation and accolades as compared to something as simple as addition of an input button with fancy </em><strong><em>css</em></strong><em> in the user interface. So if your satisfaction depends upon the outside world, you may not feel very happy and content even if you get it right after days of hard work. But otherwise the satisfaction and peace that you earn for yourself is incomparable. I feel more content, satisfied with a heightened sense of achievement after I resolve a critical investigation. This is my personal experience and it may not apply to all.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*49fcZhyfFv4O-qj1" /><figcaption>Photo by <a href="https://unsplash.com/@brucemars?utm_source=medium&amp;utm_medium=referral">bruce mars</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>An investigation also gives me a pseudo sense of being in authority where I play the detective who has just accepted a challenge to resolve the mystery that no-one has any clue about. If you have a supportive team and manager, you will be provided with all the arms and ammunition needed in the hunt, be it the much needed access to the secret vaults, databases, logs, talents and what not.</em></p><p>But this can be a very intimidating and daunting task. There may be moments, where you will find no light of hope and you may want to give up. You might have exhausted all the options, every doubtful aspects might have been ruled out, there may not be enough data points or clue to guide you any more and you may feel like you have hit a dead end. This will happen in this very case or the other, take it from me.</p><p>This is where <strong>patience</strong> and <strong>perseverance</strong> come into picture. There are no substitutes for these two qualities. You may want to take a break for sometime, get some fresh air or coffee or whatever. Sometimes with long hours of continuous focus on one subject, our creativity starts taking a back seat. There has been many times where I get ideas and clues not while I am digging the dirt but while I am in the toilet or while taking a shower or watching a film and also sometimes in my dreams. That’s our subconscious mind in action, it works in its own mysterious ways.</p><p>Everyone has their own way of performing a technical investigation. Its more about what works best for you. Here is what has worked for me on most of the occasions, not necessarily in that order.</p><ul><li><strong>Chart out sequence of events as it happened.</strong> Write them down if that works best for you. A story may immerse out of this. Take help from product team, customer facing teams etc if required to understand the episode. Every detail counts. If you know the timeline of the first occurrence of the issue, may be you can have a look at the commits in your source control like git to make sure it was not because of some new changes deployed recently.</li><li><strong>Narrow down the timeline of events. </strong>This helps in driving a focused investigation. We will look into data points within this time range, be it application logs, network logs, database entries, metrics etc.</li><li><strong>Test if the issue is reproducible locally. </strong>You can test/debug the code in local machine or your testing environment and see if you can reproduce the issue. If you can, the case is half solved. Now its just about verifying the scenario in production and planning for the fix.</li><li><strong>Collect all possible arms and ammunition.</strong> Collect whatever data points you can, be it application behavior, crucial data inputs like userID, domainID etc, timeline of the events, application logs, codes, git commits, databases, services, monitoring tools etc.</li><li><strong>Deep dive into application logs, they are the guiding stars across the dark ocean.</strong> The challenge we face with production is that we can not replay or trigger every use case and debug the execution or monitor the flow of data across layers whenever we want the way we can do it in our local machines. Knowing exactly what to look for in the log will make it more productive of course.</li><li><strong>Study data in relevant database tables or data stores. </strong>Sometimes these data can reveal the whole story. This can always be combined with the application logic to gain more insight.</li><li><strong>Eliminate what you think are normal or sane.</strong> Its not always that you are so crazy about an abnormal or insane behavior. But today, that’s exactly what you are looking for. In order to cut off the noise, I start eliminating possibilities that are unlikely to have caused the issue. This way, out of many possibilities, I have only few ones to focus on. Scope is further narrowed down.</li><li><strong>Take frequent notes.</strong> Keep on noting down any hint or clue or unusual data in your favorite editor. These may help in joining the dots later.</li><li><strong>Read and re-read the source code.</strong> Sometimes, when I feel as if I have lost it, I start afresh by reading the source code. In worst kinds of investigations,<em> I have re-read the critical sections multiple times as many as more than 5–6 times.</em> I know how that sounds. But at times it does help. Its human tendency to ignore simple and usual things. But sometimes that is where the clue lies.<br> <br>In a recent investigation, we were almost certain of the section of code that could have caused the issue but everything in that section looked so straight forward. So we shifted our focus to something else but we could not find anything helpful. Later I came back to the same section of code and re-read it twice and eventually I could locate the issue. It was a very simple thing but it did not attract our attention because that code appeared to be very simple and harmless. It had the SQL statement update messages_metadata set migrated = true where msg_id = XX. The statement was getting executed dozen of times. The issue was - the table messages_metadata had grown to contain millions of rows over last few weeks and the where clause was taking abnormally long time because the column msg_id did not have an index. This had caused performance issues.<br>Most often when you combine the code, database entries, the logs and the notes you have taken, a pattern will start emerging.</li><li><strong>Combine insights from metrics and monitoring tools.</strong> Bring in everything possible if required. If your system has tools to collect metrics of system/services, try to analyze them within the impact time range. For example, even if your focus had been on certain java code, performance insight of the database can help you with another perspective and you may end up finding some non performant query as real culprit running in entirely other part of the application. You never know.</li><li><strong>Investigate it while its fresh.</strong> Its always wise to do the investigation as soon as the issue has been reported. Prolonging it only makes life difficult as the state of the system might have changed a lot and you may have to dig through bigger data sets at every level only to find yourself deep down in sh*t.<br> <br><em>More importantly, prolonged issues are like ticking time bomb that can explode when you least expect them. If your management or leaders try to de-prioritize it, this is the time to shine by convincing them to prioritize it above anything else if you find it to be critical that needs immediate attention. Its in the interest of everyone.</em></li><li><strong>Peer-investigate.</strong> Take help from peers who can help you in the investigation. Sometimes even a small input can prove to be a game changer. Its always advisable to have another pair of eyes when you start losing hope.</li><li><strong>Share findings to stake holders. </strong>Once we have located the root cause, we should share it to the stake holders in the best way possible so that a right decision can be taken to fix the issue or put it in plan for some future release based on its severity and criticality.</li></ul><p>But then there are still some <a href="https://en.wikipedia.org/wiki/The_World%27s_Greatest_Unsolved_Crimes">unresolved mysteries and crime cases</a> in the world. Even the best of the best investigating agencies of the world have not been able to resolve them.</p><p><em>If your application does not have helpful logging, metrics tools, lacks timestamps of events in data store entries etc, you just entered into the battle without any arms and ammunition. Sometimes, it is also wise to know when to stop. Let them go in history as another unresolved issue if there is nothing much that can be done provided we left no stone unturned. At the same time, this also gives us an opportunity to know where the application lack in terms of logging, observability, monitoring and so on. Time to improve upon them so we are better equipped for future.</em></p><p><strong><em>Please feel free to provide your valuable feedback and share this story among your peers if you find this useful.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f644ae91d98" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>