<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ivan Sebastian on Medium]]></title>
        <description><![CDATA[Stories by Ivan Sebastian on Medium]]></description>
        <link>https://medium.com/@ivantedja?source=rss-c74e3b7e78d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*H_Hdjiw5gO5NDWcKRmW7Dw.jpeg</url>
            <title>Stories by Ivan Sebastian on Medium</title>
            <link>https://medium.com/@ivantedja?source=rss-c74e3b7e78d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:18:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ivantedja/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One Squad, Too Many Services: How a Small Team Survived Inheriting Everything]]></title>
            <link>https://ivantedja.medium.com/one-squad-too-many-services-how-a-small-team-survived-inheriting-everything-69ad9c46f930?source=rss-c74e3b7e78d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/69ad9c46f930</guid>
            <dc:creator><![CDATA[Ivan Sebastian]]></dc:creator>
            <pubDate>Thu, 24 Sep 2026 03:19:43 GMT</pubDate>
            <atom:updated>2026-09-24T04:25:32.283Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m_gdzjeWF3sLbYBbuSqewQ.png" /></figure><p><em>Lessons from leading a small team through a mountain of inherited code</em></p><p>One day I sat down and listed everything my squad was now responsible for. The list included more than a dozen services: Go backends, a Next.js frontend, workers, schedulers, and integrations. Some of that code was written years before any of us joined.</p><p>The git history tells you how it got there. At its peak, dozens of engineers across several teams were committing to these repositories every quarter. Today the codebase is still the same size, but the people looking after it would fit around a single table.</p><p>This is the story of how a small team learned to live with that. In the end we were more comfortable than when we started.</p><h3>From many teams to three squads</h3><p>Our area used to have many teams and many people. Then a restructuring came and left three squads. Two had clear, well-defined domains. The third was mine, and in practice it was the <em>“else”</em> squad: anything that did not belong to the other two came to us.</p><p>My team at the time:</p><ul><li>Me, as both engineering manager and tech lead</li><li>4 backend engineers</li><li>1 frontend engineer</li><li>1 QA engineer</li></ul><p>On the product side we had a single PM. That was actually good news. With only one PM, there is a natural cap on how much new product work gets pushed at once, so the roadmap had a ceiling.</p><p>The roadmap was not the problem.</p><h3>The real problem: what already exists</h3><p>What made the job daunting was not new features. It was <strong>the number of services we had to keep alive</strong>, and the fact that several of them were not mature yet:</p><ul><li><strong>Known bugs</strong> that had been sitting there for a long time.</li><li><strong>Manual operations</strong> for certain events, where an engineer had to run a script or touch the database by hand every time.</li><li><strong>Sporadic issues</strong> that only showed up months later. A service would look fine, and then one day a strange report would arrive that nobody had seen before.</li></ul><p>For the first few months I felt overwhelmed. The team kept getting pulled back and forth between building product and fighting ops fires. I will be honest: I was tempted to reject issues, or push them to someone else with “this isn’t ours” or “we don’t have context on this.”</p><p>But those services were ours now. No other team was going to take them. The only real option was to find a way to handle all of it.</p><h3>Docs help, but the code is the source of truth</h3><p>The usual advice is “read the documentation.” Documentation helped with the high-level picture. But most of the docs we inherited had gone <strong>stale</strong> over time: the code kept changing and the docs did not.</p><p>So I set one rule for the team: <strong>the code is the single source of truth.</strong></p><p>Every engineer on my team is expected to trace and debug any code we own, including code they have never seen. This matters most for products that were handed over with no knowledge transfer at all. Those usually reach us as an ops issue or a bug report.</p><p>For those cases we only need three things:</p><ol><li><strong>Where is the problem?</strong> A video or a screenshot from whoever reported it.</li><li><strong>What was expected?</strong> What should have happened instead.</li><li>With those two, <strong>start tracing</strong>: from the screen to the frontend, to the endpoint, to the backend service, to the query, to the table.</li></ol><p>There is no need to wait for someone who “knows the system.” The code knows the system.</p><h3>Champions, not experts in everything</h3><p>Tracing gets you through an unfamiliar issue. It does not give you deep knowledge. And with this many services, expecting everyone to know every feature deeply was never realistic. There is simply too much.</p><p>So I grouped our work into a few big feature areas: the learning platform, the core platform, a certification program, and a bucket for everything else. Each area got a <strong>champion</strong>.</p><p>Being a champion means:</p><ul><li><strong>You go deep on that area.</strong> You know its history, its quirks, and where the bodies are buried.</li><li><strong>You own it, even when you are not the one building it.</strong> A champion does not have to do every task in their area, but they stay aware of what is changing and keep an eye on its health.</li><li><strong>Everyone else still traces.</strong> Having a champion does not mean the rest of the team gets to say “that’s not my area.” Everyone is still expected to trace and debug anything.</li><li><strong>People help across areas.</strong> When one area gets busy, engineers from other areas jump in, and the champion is there to give them context.</li></ul><p>This worked well for backend, where there were several engineers to share the areas. It was much harder for frontend and QA. With only one person in each role, they had no choice but to know a bit of everything.</p><p>Over time the load also became uneven. Some big feature areas were far more active than others, so some champions were busier than the rest. I decided to let that be. Rebalancing for the sake of symmetry would have cost more than it saved. The goal was never equal workload. The goal was that every area had someone who truly owned it.</p><h3>Step one: know what you own</h3><p>Before I could fix anything, I needed to know exactly what we had. So I built a <strong>service catalog</strong>. It is simple:</p><ul><li>Which services are <strong>backend</strong> and which are <strong>frontend</strong></li><li>Which ones have an active <strong>deployment</strong></li><li>Where the <strong>entry points</strong> are: HTTP, gRPC, workers, cron jobs</li><li>Which <strong>databases</strong> and <strong>in-memory stores</strong> each one connects to</li></ul><p>Basically, it is a list of everything that needs watching. It is not fancy, but for the first time we had a map. When an issue came in, we knew where to start and what else might be affected.</p><h3>Every ops issue goes through one door</h3><p>Next, I needed ops issues to be <strong>visible and measurable</strong>.</p><p>I created <strong>one Slack channel</strong> with one rule: every ops issue goes through it. No DMs and no mentions buried in random threads.</p><p>I could have built Slack workflows or forms for tracking. Instead I picked the simplest thing that works: a few emoji reactions to mark the status of each issue. There is no friction, and everyone understands it right away.</p><p>More importantly, this gave me data. I could see <strong>which services broke most often, and which kinds of issues kept coming back.</strong> That data decided what we worked on next.</p><h3>Focus: eliminate ops, don’t just fight it</h3><p>With a map and some data, my priority became clear: <strong>fix ops and fix bugs.</strong></p><p>Bugs get fixed. There is nothing to debate there.</p><p>Ops improvement was the more interesting part. The principle was:</p><blockquote><em>If something requires an engineer to act manually, turn it into something the ops team can do themselves. Engineers should not have to run scripts or touch the database by hand anymore.</em></blockquote><p>That principle led to three approaches.</p><h3>1. Turn it into a feature</h3><p>Recurring manual operations became proper features in the product, with a normal flow and confirmation steps. The ops team can run them without waiting for an engineer.</p><h3>2. An Ops Toolkit for human error</h3><p>It turned out that a normal flow with confirmations still goes wrong. People mistype, pick the wrong option, and click the wrong button.</p><p>So we built an <strong>Ops Toolkit</strong>: a set of tools designed around the human errors that happened most often. With it, the ops team can <strong>fix data directly</strong>, with limited privileges and only for the cases we have already identified.</p><p>This is different from handing out database access. The toolkit can only do what it was designed to do, so it is safe, scoped, and auditable.</p><h3>3. Auto-recovery for what we don’t understand yet</h3><p>Then there is the most annoying category: issues that keep happening on a regular basis, but whose root cause we have not found.</p><p>For those, we built <strong>dedicated schedulers</strong> that automatically repair the data or the flow we used to fix by hand with a script.</p><p>I will admit this is not the textbook answer. Sometimes we simply did not have the time to dig for the root cause. But the solution <strong>works, it is pragmatic, and it is safe</strong>. The time we stopped spending on the same fix over and over went to more important things, including, eventually, going back to find the root cause without anyone breathing down our necks.</p><h3>The results</h3><p>Because every issue goes through one channel and gets tagged, I can see how the numbers move over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQfC0ydsii4rCjc109v-1Q.png" /></figure><p>In the first months after we took everything over, we averaged <strong>around 38 ops issues a month</strong>. Once the toolkit and auto-recovery were in place, that dropped to <strong>around 12 a month</strong>, roughly <strong>a 70% reduction</strong>.</p><p>The chart is not a smooth line going down, and it should not be. Every time we ship a new feature, issues go up again: a bug slips through, or a flow we did not anticipate shows up in production. You can see that in the last few months. <strong>That spike is expected.</strong> What matters is that we now have a playbook for it. We fix the bugs, turn the recurring manual fixes into toolkit actions or auto-recovery, and the numbers come back down again.</p><p>The time that used to disappear into ops went back to product development. Eventually the team only needed <strong>3 backend engineers</strong> to cover the same set of services.</p><h3>What I learned</h3><p>Here is how we survived, in short:</p><ol><li><strong>Accept reality first.</strong> Rejecting or deflecting issues does not make the services go away.</li><li><strong>The code is the source of truth.</strong> Docs are good for the big picture, but every engineer should be able to trace things themselves. Start from “where is the problem” and “what was expected.”</li><li><strong>Map everything.</strong> A simple service catalog is enough to know what you own.</li><li><strong>One door for ops, with frictionless tracking.</strong> Emoji reactions in Slack were enough to produce real data.</li><li><strong>Attack the most frequent ops issues first.</strong> Turn them into features, build an Ops Toolkit, or add auto-recovery.</li><li><strong>Being pragmatic is fine, as long as it is safe.</strong> Not every problem has to wait for a root cause.</li><li><strong>Expect spikes after releases.</strong> A spike is not a failure as long as you have a way to bring it back down.</li></ol><p>Running too many services with a small team is exhausting at the start. But it turned out we did not need more people. We needed to stop doing the same work over and over.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69ad9c46f930" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jumping Three Years of Canvas LMS in One Night: Lessons From a Big-Bang Upgrade]]></title>
            <link>https://ivantedja.medium.com/jumping-three-years-of-canvas-lms-in-one-night-lessons-from-a-big-bang-upgrade-79bee4ab559b?source=rss-c74e3b7e78d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/79bee4ab559b</guid>
            <dc:creator><![CDATA[Ivan Sebastian]]></dc:creator>
            <pubDate>Thu, 17 Sep 2026 03:19:01 GMT</pubDate>
            <atom:updated>2026-09-17T03:19:01.720Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABi2RgrZzL-QLYhiFwfHgQ.png" /></figure><p><em>How we upgraded a self-hosted Canvas LMS from July 2022 to October 2025, turned an 18-hour database migration into 5 hours, and what we’d still do differently.</em></p><p>Self-hosting an open-source platform feels great on day one. You get full control, you can customize it, and you run it on your own infrastructure. Then the months go by, upstream keeps shipping, and one day you notice your version is three years behind.</p><p>That’s where we were with <strong>Canvas LMS</strong>, the open-source learning management system from Instructure that runs our teacher learning platform. Our fork was based on a <strong>July 2022</strong> snapshot. Upstream was at <strong>October 2025</strong>. Security patches, bug fixes, and new features were piling up on the other side of a gap that got harder to cross every month.</p><p>This post covers how we crossed it: the plan, the hardest problem we hit, how we fixed it, and the one thing we’d change.</p><h3>Why a three-year gap is harder than it sounds</h3><p>Most of the application code merged without much trouble. The database was the hard part.</p><p>Canvas upstream <strong>squashes old database migrations about once a month</strong>. Older migration files get merged into one consolidated file, and the originals disappear from the repo. That keeps things tidy for teams that upgrade often. For a team that’s three years behind, it’s a trap.</p><p>In the latest upstream code, the oldest unsquashed migration was from around April 2025. So db:migrate only works if your database is already newer than that. Ours was from 2022. To get from our schema to the new one, we had to <strong>un-squash almost three years of migrations</strong>, which meant:</p><ul><li><strong>Reverting many squash commits</strong>. Squashing happened roughly monthly, so there were a lot of them to undo.</li><li><strong>Dealing with helpers that changed underneath the migrations</strong>. Some migrations import shared helper files. After the old migrations were squashed away, those helpers were updated for the newer migrations, and the updates didn’t work with the old ones we’d restored. We fixed this by adding copies of the old helper versions and pointing the restored migrations at them by hand.</li><li><strong>Fixing migration state manually</strong>. A few indexes had to be renamed and some migration versions inserted straight into schema_migrations so the migrator would agree with what was actually in the database.</li></ul><h3>The strategy: no rollback, just a restore point</h3><p>With three years of schema and data changes, a normal rollback plan wasn’t realistic. Reversing hundreds of migrations, some of which change data rather than just structure, would be riskier than the upgrade.</p><p>So we made a clear call: <strong>no rollback plan. We’d rely on a restore point instead</strong>.</p><ol><li>Take a full backup (clone) of the production database.</li><li>Run the migrations.</li><li>Deploy the new version.</li><li>If something goes seriously wrong, <strong>restore the backup</strong> and redeploy the old version.</li></ol><p>The trade-off was explicit and written into the plan: restoring means losing any data created on the new version. That’s why the order of steps and the internal testing window mattered so much.</p><p>“No rollback” doesn’t mean “no caution.” We made up for it by testing heavily beforehand.</p><h3>Preparation: three months on staging</h3><h3>A separate deployment and URL on staging</h3><p>We didn’t upgrade staging in place. We set up a <strong>separate deployment with its own URL</strong> for the new Canvas version, running alongside the old one. That let us:</p><ul><li>Compare old and new behavior side by side</li><li>Run regression tests without blocking other teams on staging</li><li>Throw it away and rebuild it as many times as we needed</li></ul><h3>Letting it soak</h3><p>The new version ran on staging for <strong>about three months</strong> before we went near production. Integrations, authentication, course flows, and background jobs all got real use during that time. When we set a production date, we were confident the application worked. The remaining unknowns were about running the migration against production data.</p><h3>A step-by-step checklist</h3><p>Everything went into a written runbook, down to individual commands. It covered:</p><p>Before the maintenance window</p><ul><li>Enable required Postgres extensions on production (the new version needs pgvector)</li><li>Grant the migration job’s service account the permissions it needs (for example, Pub/Sub access)</li><li>Set up CI/CD variables for the production migration and deploy jobs</li><li>Update authentication provider config to match the identity provider’s token issuer</li><li>Run manual SQL to fix migration state</li><li>Prepare the maintenance page</li><li>Set up an internal test domain pointing at the production service</li></ul><p>During the window</p><ol><li>Create the release tag and let the automatic pipeline jobs finish, keeping migration, asset upload, and deploy as manual jobs</li><li>Turn on the maintenance page</li><li>Stop all existing Canvas services (scale to zero, disable autoscaling schedulers)</li><li><strong>Clone the production database as a backup</strong> and wait for it to finish</li><li>Run the migration and asset upload jobs by hand, then deploy</li><li>Run internal testing on the test domain</li><li>Switch the domain config to the real production domain and redeploy</li><li>Turn off the maintenance page</li></ol><p>After</p><ul><li>Remove the temporary test domain and the old staging instance</li></ul><p>We also set up standby shifts with named engineers for each part of the window, and a monitoring dashboard for load balancer traffic and errors.</p><h3>A planned 24-hour downtime, on a holiday</h3><p>We planned for <strong>24 hours of downtime</strong>, covering the database migration, deployment, and internal testing. That’s a long time, so:</p><ul><li>We <strong>told all stakeholders</strong> <strong>well ahead</strong> of time</li><li>We picked a <strong>public holiday</strong> so as few teachers as possible would be affected</li></ul><h3>The hardest part: a database migration that took 18 hours</h3><p>The migration was the biggest risk, both to test and to run.</p><h3>The pipeline got stuck</h3><p>Our first full migration runs happened in the CI pipeline, and <strong>the pipeline hung</strong>. No useful error, no progress, nothing to go on.</p><p>Instead of guessing from CI logs, we <strong>provisioned a dedicated VM</strong> to run the migration by hand. That gave us a shell, the ability to watch the process live, and a fast way to try fixes.</p><p>The cause turned out to be a <strong>schema cache we had built ourselves</strong>. Earlier, to handle a big traffic spike, we patched Canvas so ActiveRecord’s primary key lookups were cached in Redis. That cut database CPU usage roughly in half (the full story is in <a href="https://medium.com/inadigital-edu/ensuring-lms-ruang-gtk-readiness-for-nationwide-digital-learning-expansion-d09f51067f93">Ensuring LMS Ruang GTK Readiness for Nationwide Digital Learning Expansion</a>). The patch deliberately had no automatic invalidation, because in our case the database schema almost never change in normal operation.</p><p>A three-year migration is not normal operation. Tables were created, renamed and restructured, and the migration kept reading stale schema metadata from Redis. Once we took that cache out of the picture for the migration run, it ran all the way through. The lesson: a shortcut that’s safe in production can bite during a migration, so write down the assumptions behind it.</p><h3>Then the real problem: 18 hours</h3><p>It completed, but on a clone of production it took <strong>about 18 hours</strong>.</p><p>That wasn’t acceptable. With 18 hours for the migration alone, plus deployment, internal testing, and the possibility of needing to restore, a 24-hour window was gone. There was no room for anything to go wrong.</p><h3>Applying 80/20</h3><p>We didn’t try to speed up every migration. We <strong>profiled the run and found the few migrations using most of the time</strong>. The Pareto principle held: a small number of migrations accounted for most of the duration.</p><p>When we read the slowest ones, we found something interesting. Some data migrations had <strong>built-in throttling</strong>: they processed rows in small batches with deliberate pauses. Upstream does this for good reason. Teams that upgrade with <strong>zero downtime</strong> have to migrate while real users hit the database, and throttling keeps the migration from hurting live traffic.</p><p><strong>We weren’t doing zero downtime</strong>. The site was behind a maintenance page with no users. The throttling was only making us wait.</p><p>Looking closer, we found that <strong>one of the heaviest migrations could be skipped entirely</strong> in our case. It handled a data situation that didn’t apply to our deployment.</p><h3>From 18 hours to 5 hours</h3><p>After removing the unnecessary throttling and skipping the irrelevant migration, production-clone runs dropped from <strong>about 18 hours to about 5 hours</strong>. For comparison, the same migration took around 47 minutes on staging’s smaller dataset, a reminder that staging timings tell you little about production.</p><h3>Knowing when to stop</h3><p>We probably could have gotten it lower. More profiling, more batching changes, maybe some parallelism.</p><p>We stopped anyway. <strong>Five hours fit comfortably in the 24-hour window</strong>, with room for deployment, testing, and a restore if needed. Every extra hour saved would cost more engineering effort for less benefit, so we moved that effort to other upgrade work. The aim was a safe upgrade, not the fastest possible migration.</p><h3>Upgrade day</h3><p>During the maintenance window we used a <strong>dedicated internal domain</strong> to test the new version against real production data for <strong>half a day</strong>, before any teacher could reach it. That was where the restore plan would have kicked in if we’d found anything serious.</p><p>We didn’t need it. <strong>About 95% of the upgrade went exactly as planned</strong>. The migration ran in the expected time, the deployment worked, and core flows passed regression testing.</p><h3>The 5%: cross-domain problems</h3><p>The one issue we hit came from the testing domain.</p><p>Our internal test domain was on a <strong>different parent domain</strong> from production. Canvas and the services around it depend on cookies and cross-origin requests, so testing on a separate domain caused:</p><ul><li><strong>CORS errors</strong>, because the test origin wasn’t allowed where the production origin was</li><li><strong>Cookie problems</strong>, because cookies scoped to the production domain weren’t sent to (or set from) the test domain</li></ul><p>None of this was a real application bug, but it cost time during a window where time mattered, and it made it harder to tell real problems apart from test-setup artifacts.</p><p><strong>What we should have done</strong>: create the test subdomain <strong>under the same parent domain</strong> as production. Cookies and CORS would have behaved the same as in production, and testing would have been smooth.</p><p>We knew this was the better option. It fell through when plans changed. The maintenance was rescheduled, and in the shuffle, setting up that domain was dropped. Looking back, <strong>the reschedule gave us enough time</strong> <strong>to do it</strong>, and we didn’t use it. When a plan changes, go back over the items you deferred, because the new timeline may have room for them.</p><h3>Key takeaways</h3><p>1. <strong>Don’t let upstream get three years ahead</strong>. Most of the pain came from one thing: upstream squashing migrations monthly while we stood still. We now plan to <strong>upgrade about every three months</strong>, so our schema never falls behind the oldest unsquashed migration.</p><p>2. <strong>When rollback isn’t realistic, commit to a restore point and test hard</strong>. “No rollback” was the honest choice. We backed it with a full database clone, three months on staging, and a half-day internal test before users came back.</p><p>3. <strong>Run the migration against a production-sized clone</strong>. Staging took 47 minutes. The production clone took 18 hours before optimization. Only production-sized data will show you that.</p><p>4. <strong>Get off CI when you need to debug</strong>. A stuck pipeline tells you almost nothing. A VM with a shell showed us the schema cache problem quickly.</p><p>5. <strong>Question upstream’s assumptions for your situation</strong>. Throttling makes sense for zero-downtime upgrades and wastes time during planned downtime. Some migrations don’t apply to you at all. Read the slow ones before you try to optimize them.</p><p>6. <strong>Use 80/20 and know when to stop</strong>. Fix the few things that eat most of the time. Once you’re within budget, move people to other work.</p><p>7. <strong>Make your test environment match production, including domains</strong>. Cookies and CORS depend on the domain. A test subdomain under the production domain would have avoided our only real problem.</p><p>8. <strong>When a plan is rescheduled, revisit what you deferred</strong>. New timelines often free up time for the things you’d dropped.</p><p>Upgrading across three years in one maintenance window wasn’t glamorous, but it worked: a clear strategy, a thorough checklist, and practical engineering where it mattered. If you’re running a self-hosted Canvas (or any fast-moving open-source platform) that’s fallen behind, we hope this helps you plan your own upgrade. And if you can, upgrade often so you never need a post like this.</p><p><em>Have you done a big-bang upgrade on a self-hosted platform? I’d like to hear how you handled it. Share in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79bee4ab559b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Diversity and Inclusion in an Organization. Real or Fake?]]></title>
            <link>https://ivantedja.medium.com/diversity-and-inclusion-in-an-organization-45188143ed7d?source=rss-c74e3b7e78d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/45188143ed7d</guid>
            <category><![CDATA[workplace]]></category>
            <category><![CDATA[diversity]]></category>
            <category><![CDATA[inclusion]]></category>
            <dc:creator><![CDATA[Ivan Sebastian]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 15:51:38 GMT</pubDate>
            <atom:updated>2022-04-04T14:10:38.994Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Disclaimer: This post is my personal opinion. I don’t have any intention to offend or insult any individual, group of people or organizations</strong></blockquote><p>Today I heard a great sharing in my workplace regarding the impact of diversity and inclusion in an organization. There are many articles discussing this topic, I’m not gonna delve too deep on the topic.</p><p>In general, I agree that diversity and inclusion can give a good impact on an organization. As employees, they usually feel more happy and motivated which eventually leads to better performance to the individual and organization as a whole.</p><p>This topic brought back my memories to a conference that I attend years ago. There were two speakers who shared about their organizations. They shared about diversity and inclusion in their organizations, how they hire people from all around the world regardless of their background. At some point, there were very interesting conversations that I heard.</p><p>I couldn’t remember the exact wording, but one of the speakers said that they intend to be more diverse by hiring more people from different backgrounds. At that year, they aim to reach 50% on diversity (I’m purposely not mentioning what kind of diversity was aimed).</p><p>Then the other speaker said: it means yours are not diverse? Because you tried to hire people from specific backgrounds to reach that number of diversity.</p><p>Boomm! I was really surprised by that thought. Up until now, I do agree with that thought. It‘s very cliche. An organization aims to reach diversity and inclusion through “un-diverse” and “un-inclusive” ways by purposely hiring specific backgrounds.</p><p>Let’s use an example. Let’s say there is only 1 vacant position in a company. Then there are 2 candidates that are equally good based on competency. One is a woman, and the other one is disabled. The company already has a lot of women employees and doesn’t have many disabled. The company aims to be more diverse by hiring more disabled people. So the 1 quota is given to the disabled candidate. How is that fair? How is that diverse for the woman?</p><p>In my opinion, it’s true that achieving diversity is not easy. By nature, minority obviously lower in number. Of course, it is hard for an organization to be more diverse by having more minorities <strong>organically</strong>. An organization that supports diversity and inclusion surely has more applicants from minorities. So the organization can be more diverse. But still, achieving a specific diversity number <strong>organically</strong> doesn’t make sense for me. Unless the organization <em>cheatingly</em> accepts people from specific backgrounds.</p><p>The goal is great — to support more minorities. I really support that. I also support if an organization <strong>admittedly</strong> tried to achieve a specific diversity number through “un-diverse” and “un-inclusive” ways. Because I know, it is not easy but the impact of having more diverse people is great.</p><p>Right now, the only <strong>organic</strong> way that I can think of to reach a specific diversity number is by creating a specific product that is targeted to <strong>minorities only</strong>. Then make those minorities love their product and want to work with them. But.. which company can have such privilege to do that.</p><p>Just my 2 cents :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45188143ed7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Investigating Sinatra + Active Record + Cloud SQL Connection Bottleneck]]></title>
            <link>https://ivantedja.medium.com/investigating-sinatra-active-record-cloud-sql-connection-bottleneck-c97ff2bde02f?source=rss-c74e3b7e78d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c97ff2bde02f</guid>
            <category><![CDATA[cloud-sql]]></category>
            <category><![CDATA[activerecord]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[sinatra]]></category>
            <dc:creator><![CDATA[Ivan Sebastian]]></dc:creator>
            <pubDate>Fri, 02 Apr 2021 16:47:55 GMT</pubDate>
            <atom:updated>2026-06-21T15:44:56.072Z</atom:updated>
            <content:encoded><![CDATA[<h3>How a One-Line Fix Cut Our Database Connections by 40%</h3><blockquote><strong>A database connection graph that stayed perfectly flat at 3 a.m. — exactly when traffic should have been at its lowest — led me to a misconfiguration that had been hiding in production for over a year. Here’s how I tracked it down.</strong></blockquote><p>At <a href="https://en.wikipedia.org/wiki/Bukalapak">Bukalapak [1]</a> we had hundreds of microservices deployed on Google Cloud, running using <a href="https://kubernetes.io/">Kubernetes [2]</a>. Our databases mostly used <a href="https://cloud.google.com/sql/docs">Cloud SQL [3]</a>. Our code base was mostly written in Ruby and Golang.</p><p>Here I’m gonna share my experience when investigating bottleneck on one of the microservice that was managed by my <em>Squad</em>. The microservice was powered by <a href="http://sinatrarb.com/">Sinatra [4]</a> and used Cloud SQL for MySQL <em>(I will just refer to it as Cloud SQL for the rest of this article)</em>.</p><blockquote>In Bukalapak, we call the team as Squad: which usually consists of Product Manager, Engineering Manager, Test Engineering Manager, Backend Engineer, Frontend Engineer, Mobile Apps Engineer, Test Engineer, Data Scientist, Designer</blockquote><p>The Cloud SQL has a default <a href="https://cloud.google.com/sql/docs/mysql/quotas#maximum_concurrent_connections">4000 concurrent connection limit [5]</a> for the machine type that we used. This limit can be configured but was not recommended as exceeding the default limit might cause instability.</p><blockquote>You can use the <a href="https://cloud.google.com/sql/docs/mysql/flags#mysql-m">max_connections</a> flag to configure connections limits. The Cloud SQL team recommends using the default connection limits to ensure instance stability. You can find the connection limits for your instance by connecting to your database and running this command: SHOW VARIABLES LIKE &quot;max_connections&quot;;</blockquote><h3>About The Microservice</h3><p>Our microservice mostly handled interactions with third party. So there was a lot of external API call (means it involves the internet). As for the tech stacks / services involved were:</p><ul><li>Ruby using Sinatra framework</li><li><a href="https://github.com/puma/puma">Puma [6]</a> for the web server</li><li><a href="https://stackoverflow.com/a/36278672/1363015">Active Record [7]</a> for the <a href="https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping">ORM [8]</a></li><li><a href="https://www.datadoghq.com/">Datadog [9]</a> for the monitoring service</li><li><a href="https://www.atlassian.com/software/opsgenie">OpsGenie [10]</a> for the alerting service</li><li>Kubernetes with <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">HPA (Horizontal Pod Autoscaler) [11]</a> configured</li></ul><h3>The Problem?</h3><p>As mentioned previously, we had a limit of 4000 connections to the Cloud SQL. As our Squad’s features grew, our microservice received more and more traffic. At that time it can reach ~2600 rps (request per second).</p><p>As one of the biggest marketplace in Indonesia, it was very normal to have a big promotion / campaign. Our microservice’s features were located in a lot of critical places (such as homepage, checkout page, etc). An increase in overall Bukalapak traffic, most of the time would contribute to the increase of our microservice’s traffic.</p><p>Luckily (or not?) our microservice had a good monitoring and alerting system for the applications (throughput, latency, error rate, cpu, memory, etc) and the database (cpu, memory, number of active connections, etc).</p><p>We realize that our microservice had a high number of active database connections (~1300 connections). It’s still safe for business as usual, but it might hit the 4000 connection limit bottleneck if the traffic surge by 2x or 3x 🔥(which may happen for big campaign with live TV show).</p><h3>Abnormalities</h3><p>One day, in a bright sunny day 😎 , I spent some time looking for clues in the Datadog metric for Database Connections. I saw some weird pattern at dawn where our microservice’s traffic was at its lowest.</p><p>I saw a flat pattern. It’s <strong>literally</strong> a <strong>flat</strong> line without any dip from ~0am-8am. My intuition said I should see the metric go down at ~3am, correlating with our lowest throughput each day.</p><p>I had no idea what’s going on. I knew the general concept of a connection pool, but I didn’t know the implementation details of how Active Record handled database connections.</p><h3>Investigations</h3><p>I then began my investigations related with Sinatra + Puma + Active Record + MySQL. I tried to understand how the Database Connection was being used by Active Record.</p><h4>Start Small</h4><p>Our microservice had dozens of libraries installed and a lot of business logics. I thought that it might not be the right place to use as a playground. There might be custom behavior caused by third party libraries. So I decided to <strong>start small</strong>, by creating a simple Sinatra application with Puma web server, then observe how it worked.</p><p>Here is what my setup looked like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eGhzgF8lRYjAzm0HRH8LMA.png" /><figcaption>Simple Playground Application</figcaption></figure><p>I’m not copy-pasting any code in here, as there was nothing fancy about the playground application:</p><ul><li>I created a MySQL database with 1 table and 1 record <em>(this step should be able to be skipped and replaced by using raw SQL query </em><em>SELECT 1 , but at that time I didn’t do that to closely replicate the condition of our microservice)</em></li><li>Sinatra Application that used Puma web server, with Active Record for the ORM</li><li>The playground application exposed 2 endpoints: one endpoint /0 accessed the database, and the other one /1 did not access the database. I used puts on the Thread ID to debug the application</li><li>Active Record was configured with a connection pool of 2</li></ul><p>I ran the application using 2 puma Threads with following command:</p><pre>$ puma -t 2:2</pre><p>I then hit it with 8 concurrent requests</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*u2YTDp0umWrqeVDJj5R6zg.gif" /><figcaption>Database Connection Pool Equals with Puma Threads</figcaption></figure><p>Everything looked fine, I didn’t notice any issue at all. Then I did another experiment, setting the database connection pool to a lower number than the Puma Thread. In our case, I set the database connection pool into 1</p><pre>ActiveRecord::Base.establish_connection(<br>  adapter: &quot;mysql2&quot;,<br>  database: &quot;goods&quot;,<br>  username: &quot;root&quot;,<br>  password: &quot;root&quot;,<br>  host: &quot;127.0.0.1&quot;,<br>  port: 3306,<br>  pool: 1, # &lt;--- I changed it from 2 into 1<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*ZsMIpYaqSLcUIS9kUFdUFw.gif" /><figcaption>Database Connection Pool Lower than Puma Threads</figcaption></figure><p>BOOM! 💥 We saw the infamous error: <strong><em>could not obtain a connection from the pool within 5.000 seconds (waited 5.005 seconds); all pooled connections were in use</em></strong></p><p>Here is the screenshot version:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ws4yeu4KiUuWnDuSBJr7ug.png" /><figcaption>Error: Could Not Obtain a Connection</figcaption></figure><p>We can see that there was 8 requests, I’ll sum it up along with the Thread ID(s):</p><pre>+---------+----------------+----------+<br>| Request |   Thread ID    | Endpoint |<br>+---------+----------------+----------+<br>|       1 | 70210596097360 | /1       |<br>|       2 | 70210596097060 | /1       |<br>|       3 | 70210596097060 | /0       |<br>|       4 | 70210596097060 | /1       |<br>|       5 | 70210596097060 | /0       |<br>|       6 | 70210596097060 | /1       |<br>|       7 | 70210596097060 | /1       |<br>|       8 | 70210596097360 | /0       | &lt;--- Error<br>+---------+----------------+----------+</pre><p>I ran the test several times and I saw that: a request to endpoint /0 was only succeded on a specific Thread ID. On example above, it was to Thread ID 70210596097060</p><h4>Research</h4><p>I tried to search in the internet regarding this behavior, and found this post <a href="https://stackoverflow.com/a/36278672/1363015">https://stackoverflow.com/a/36278672/1363015 [12]</a>. Quoting the important part:</p><blockquote>This method (as described in the <a href="https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb#L254">source code docs</a>) returns the connection <strong>associated with the current thread</strong>. Internally it uses a connections cache where the key is the Thread.current.object_id, i.e. a unique ID of the current thread.</blockquote><p>It appeared that a Puma Thread will be mapped as a key to a database connection. As seen in my previous experiment, only Thread ID 70210596097060 could use the database connection. The other Thread ID 70210596097360 could not obtain the database connection.</p><blockquote>The link was pointing to Active Record version 4.2. There was some changes (my playground application used the latest Active Record: version 6.1.3.1) since then, but the general logic was still similar.</blockquote><p>I took it for granted. I then thought that setting database connection pool lower than Puma Thread would cause a disaster (like what happened on my playground application)!</p><p>I discussed that finding and my experiment with my colleague Yap Sok Ann. My colleague found that to be weird and should not be like that. I was suggested to debug the checkout and checkin method in lib/active_record/connection_adapters/abstract/connection_pool.rb.</p><p>There were several ways to do this, such as:</p><ul><li>Directly insert puts to the Active Record source code</li><li>Use a debugger if your IDE supports it (my favorite is RubyMine)</li><li>Or just directly patch the code. I’ll use this one for this article.</li></ul><p>I inserted this code to app.rb:</p><pre>module <em>ActiveRecord<br>  </em>module <em>ConnectionAdapters<br>    </em>class <em>ConnectionPool<br>      </em>alias original_checkout checkout<br>      def checkout<br>        puts &quot;___checkout: #{Thread.current.object_id}&quot;<br>        original_checkout<br>      end<br><br>      alias original_checkin checkin<br>      def checkin(<em>conn</em>)<br>        puts &quot;!!!checkin: #{Thread.current.object_id}&quot;<br>        original_checkin(<em>conn</em>)<br>      end<br>    end<br>  end<br>end</pre><p>Here was the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EsbxQ0DBY2_7hbgJ4lagFA.png" /></figure><p>We can see there were 2 <strong>checkouts </strong>(obtaining a connection from the pool), but no <strong>checkin </strong>(returning a connection to the pool).</p><p>If I directly call ActiveRecord::Base.clear_active_connections! <em>(I found that from this documentation </em><a href="https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html"><em>https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html [13]</em></a><em>) </em>then I could see the <strong>checkin </strong>printed. Which means my patch worked, and the <strong>checkin</strong> was truly not being called.</p><p>Suddenly… <strong>voila!!</strong> 🎉 My colleague ping me with a bug issue link: <a href="https://github.com/rails/rails/issues/37092">https://github.com/rails/rails/issues/37092 [14]</a>. It appeared that Janko had already reported it and added a workaround for Sinatra. It was to use gem &#39;sinatra-activerecord’ which would <strong>checkin</strong> the connection after a request cycle was completed <a href="https://github.com/sinatra-activerecord/sinatra-activerecord/blob/0f079e6000d2b3b00c642ab9af971466e401ba90/lib/sinatra/activerecord.rb#L33">https://github.com/sinatra-activerecord/sinatra-activerecord/blob/0f079e6000d2b3b00c642ab9af971466e401ba90/lib/sinatra/activerecord.rb#L33 [15]</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CvH1Eivne-B1wzVvVum03Q.png" /></figure><h4>Let’s Test It!</h4><p>I re-ran my application after making the following changes:</p><ul><li>In Gemfile, I changed gem &#39;activerecord’ into gem &#39;sinatra-activerecord&#39;</li><li>In app.rb, I changed require &#39;active_record&#39; into require &#39;sinatra/activerecord&#39;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*2p5B5CoFE2mzEv3QIDyKhQ.gif" /><figcaption>Now We Can See The Checkin</figcaption></figure><p>Hurraay! 💯 I could see the <strong>checkin</strong> and I didn’t see the previous error <strong>“<em>could not obtain a connection from the pool within 5.000 seconds”</em></strong>. We can also see that two different Thread IDs (70096363781900 and 70096363781620) could still access the database.</p><p>We need to be aware though, the database connection was returned to the pool <strong>after</strong> the end of the request. So if there is slowness after <strong>checkout</strong> the <strong>checkin</strong> can get delayed as well.</p><pre>request is coming<br>call database  &lt;-- checkout (obtaining database connection)<br>slow API call  &lt;-- it will cause the connection to be hold<br>end of request &lt;-- checkin (connection is returned to pool)</pre><h4>Yet Another Issue</h4><p>Pheew.. I finally found the workaround. Let’s try to use gem sinatra-activerecord for our microservice. But… it appeared that our microservice already used thesinatra-activerecord gem.</p><p>I was so confused. I checked the sinatra-activerecord‘s code and the app.after{ ActiveRecord::Base.clear_active_connections! } was there.</p><p>Then, I looked through the GitHub issues with keyword not working(yes, one by one): <a href="https://github.com/sinatra-activerecord/sinatra-activerecord/search?q=not+working&amp;type=issues">https://github.com/sinatra-activerecord/sinatra-activerecord/search?q=not+working&amp;type=issues [16]</a> in case it had been reported as well.</p><p>Wohooo!! I found similar issue <a href="https://github.com/sinatra-activerecord/sinatra-activerecord/issues/2">https://github.com/sinatra-activerecord/sinatra-activerecord/issues/2 [17]</a>. Quoting the solution:</p><blockquote>Actually I realized the app itself wasn’t working, and poking through the specs I figured out that I needed to add register Sinatra::ActiveRecordExtension to my app class to make things work. You might want to make a note of that in your read me?</blockquote><p>Danngg! It appeared that I didn’t understand Sinatra well enough. I didn’t realize that Sinatra had <strong>Modular and Classic Style </strong>( <a href="http://sinatrarb.com/intro.html">http://sinatrarb.com/intro.html [18]</a>, find: Modular vs. Classic Style). The one that I used in this article was Classic Style (that’s why it worked 🤷‍♂️), whereas the one I used in our microservice was the Modular Style.</p><p>It appeared that there was a misconfiguration in our microservice. It had been happening for over a year, since the microservice was first deployed to production.</p><blockquote>Actually this seems to be a common issue (or not?) on Sinatra application. I saw another example: <a href="https://github.com/DataDog/dd-trace-rb/pull/144">https://github.com/DataDog/dd-trace-rb/pull/144 [19]</a></blockquote><h3>Results</h3><p>I made one liner changes to our microservice by adding:</p><pre>register Sinatra::ActiveRecordExtension</pre><p>Then deployed the changes to production.</p><p>The results were striking: we reduced database connections by ~40%!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OBBUkyrzx9ITbR6p2jCmlA.png" /></figure><blockquote>Actually the metric above was not directly caused by the <strong>checkin</strong> (returning databases connection to pool). That metric was caused by <a href="https://github.com/rails/rails/pull/31221">flushing idle database connection [20]</a>. So, what actually happened was: there was a <strong>checkin</strong> then if the connection was idle for more than 5 minutes (the default idle_timeout), the connection will be flushed (disconnected). Before that commit, there might be a <strong>checkin</strong> but we couldn’t see any metrics that correlate with that.</blockquote><p>We no longer worry for the 4000 connections limit. We also did more improvement by caching some database calls which can further reduce the database connections.</p><p>Our microservice also survived peak traffic during a big campaign with throughput of ~6100 rps. The database connection was not a bottleneck anymore as we only used ~1650 database connections 🥳</p><p>In the end, it was a miss implementation / configuration from our team. But I think the journey to figure out the issue and the knowledge gained were pretty interesting.</p><h3>What I learned</h3><ul><li><strong>Start small.</strong> Reproducing the behavior in a minimal playground — away from dozens of libraries and business logic — is what made the root cause visible.</li><li><strong>Don’t take answers at face value.</strong> A StackOverflow answer sent me down the wrong path; verifying it myself by patching checkout/checkin is what corrected it.</li><li><strong>Understand your framework’s modes.</strong> A one-line difference between Sinatra’s Classic and Modular styles hid in production for over a year.</li><li><strong>Good monitoring pays off.</strong> None of this would have surfaced without a metric clear enough to reveal an anomaly at 3 a.m.</li></ul><h3>References</h3><ol><li><a href="https://en.wikipedia.org/wiki/Bukalapak">https://en.wikipedia.org/wiki/Bukalapak</a></li><li><a href="http://kubernetes.io/">http://kubernetes.io/</a></li><li><a href="https://cloud.google.com/sql/docs">https://cloud.google.com/sql/docs</a></li><li><a href="http://sinatrarb.com/">http://sinatrarb.com/</a></li><li><a href="https://cloud.google.com/sql/docs/mysql/quotas#maximum_concurrent_connections">https://cloud.google.com/sql/docs/mysql/quotas#maximum_concurrent_connections</a></li><li><a href="https://github.com/puma/puma">https://github.com/puma/puma</a></li><li><a href="https://guides.rubyonrails.org/active_record_basics.html">https://guides.rubyonrails.org/active_record_basics.html</a></li><li><a href="https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping">https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping</a></li><li><a href="https://www.datadoghq.com/">https://www.datadoghq.com/</a></li><li><a href="https://www.atlassian.com/software/opsgenie">https://www.atlassian.com/software/opsgenie</a></li><li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a></li><li><a href="https://stackoverflow.com/a/36278672/1363015">https://stackoverflow.com/a/36278672/1363015</a></li><li><a href="https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html">https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html</a></li><li><a href="https://github.com/rails/rails/issues/37092">https://github.com/rails/rails/issues/37092</a></li><li><a href="https://github.com/sinatra-activerecord/sinatra-activerecord/blob/0f079e6000d2b3b00c642ab9af971466e401ba90/lib/sinatra/activerecord.rb#L33">https://github.com/sinatra-activerecord/sinatra-activerecord/blob/0f079e6000d2b3b00c642ab9af971466e401ba90/lib/sinatra/activerecord.rb#L33</a></li><li><a href="https://github.com/sinatra-activerecord/sinatra-activerecord/search?q=not+working&amp;type=issues">https://github.com/sinatra-activerecord/sinatra-activerecord/search?q=not+working&amp;type=issues</a></li><li><a href="https://github.com/sinatra-activerecord/sinatra-activerecord/issues/2">https://github.com/sinatra-activerecord/sinatra-activerecord/issues/2</a></li><li><a href="http://sinatrarb.com/intro.html">http://sinatrarb.com/intro.html</a></li><li><a href="https://github.com/DataDog/dd-trace-rb/pull/144">https://github.com/DataDog/dd-trace-rb/pull/144</a></li><li><a href="https://github.com/rails/rails/pull/31221">https://github.com/rails/rails/pull/31221</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c97ff2bde02f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ruby Kaigi 2018: Avoiding Cognitive Overload]]></title>
            <link>https://ivantedja.medium.com/ruby-kaigi-2018-avoiding-cognitive-overload-e646e071b478?source=rss-c74e3b7e78d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e646e071b478</guid>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Ivan Sebastian]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 13:37:12 GMT</pubDate>
            <atom:updated>2018-07-21T13:37:12.669Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Khp_6iuFxCtzdVUC4eSvfQ.png" /></figure><p>I’d like to share a great talk by <a href="https://github.com/jules2689/">Julian Nadeau</a>, a Production Engineer from Shopify. The title is <strong>Avoiding Cognitive Overload: Test for Productivity and Education</strong>. <a href="https://jnadeau.ca/presentations/rubykaigi2018/">This</a> is the link to his original presentation</p><h3>Why?</h3><p>There were a lot of great talks by great speakers in Ruby Kaigi 2018. In my opinion, Julian’s was one of the best talk. Because it was easy to understand, simple, and left a deep impression to me. This talk brought back my memories when I joined Bukalapak for the first time. It reminded me how I struggled with onboarding moments.</p><h3>What?</h3><p>It started from mistakes. People make mistakes. No matter who it is, inexperienced or even the most experienced developers make mistakes. Mistakes are bad. We want to avoid mistakes as much as possible.</p><p>We need some way to avoid mistakes. Some companies has rules or guidelines to follow to avoid mistakes. As a new employee, onboarding times is one of the most difficult things (at least for me, how about you?). I didn’t want to break things on my first day. Okay, so what should I read to avoid mistakes? Of course we have documentation for engineer’s onboarding.</p><p>Well, it was very helpful. It gave a lot of informations, such as: architecture of Bukalapak, how to setup the codebase, several important checklists, etc. For me, first pull request was daunting but thankfully it went smooth. Then bigger tasks were coming.</p><p>Files changed grew bigger, pull request’s reviews keep coming. I made mistakes. I should have read about that in “page A”. Ah, for the other mistakes I should have read it in “page B”. Ahh yes, I forgot about that one. Oh okay, my colleague told me about that a month ago …</p><p>That was what Julian called as <strong>Cognitive Overload</strong>:</p><blockquote>Being given too much information all at once, resulting in being incapable of processing the information efficiently or at all.</blockquote><p>These are some examples:</p><ul><li>Don’t add gem foo</li><li>Don’t forget to add test files</li><li>This class should only have one public method</li><li>Use verb as class name</li><li>etc</li></ul><p>Those lists will grow bigger and bigger as your company grows. I’m not sure if you can remember those things unless you have a photographic memory.</p><h3>How?</h3><p>Okay, now we know what is the meaning of <strong>Cognitive Overload</strong>. Now, how do we avoid it?</p><p>Julian gave some solutions:</p><h4>* Unit Test</h4><p>Some of you might be familiar with unit test. It is used to test some logic in your code. So we assert an input to match expected output. How about using your codebase as the input?</p><p>For example:</p><ul><li><strong>Rule</strong>: some class must use verb as class name.<br><strong>Solution</strong>: We can create a test to iterate the codebase, retrieve the file name, then match it with <a href="https://developer.oxforddictionaries.com/">online dictionary</a>.</li><li><strong>Rule</strong>: in Bukalapak’s code base, some class need to have <a href="https://github.com/egonSchiele/contracts.ruby">type checking</a> in the <em>initialize</em> method.<br><strong>Solution</strong>: For Ruby, we can create a test to read the file and parse it using Ruby’s Ripper API or using <a href="https://github.com/whitequark/parser">Parser</a> to parse the file and check if the type checking is in place or not.</li><li><strong>Rule</strong>: this class should only have one public method<br><strong>Solution</strong>: well, the idea is similar as above.</li></ul><h4>* Style Guides</h4><p>For Ruby, you can use <a href="https://github.com/rubocop-hq/rubocop">Rubocop</a>. It is a static code analyzer. Enough with debate of which style is better or easier to read, just use Rubocop and we can have consistent style guide in our codebase. Now, we can spend our time to review business logics rather than reviewing style guide.</p><h4>* Just in Time Education</h4><p>Means to provide education / information at the exact moment the user needs it, not before. It can reduce <strong>Cognitive Overload</strong> as you are given the information at the exact time. You don’t need to remember a bunch of stuffs at once.</p><h4>* Other Solutions</h4><p>Give a clear error messages so it is easier to understand. Some package managers give nasty errors which looks confusing and daunting. Julian wrote a bundler plugin to make gem installation more actionable, educative, and all around easier to understand. Check <a href="https://github.com/jules2689/extended_bundler-errors">this</a> out and contribute!</p><blockquote>That’s all from me. Thanks for reading!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e646e071b478" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>