<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by saurav omar on Medium]]></title>
        <description><![CDATA[Stories by saurav omar on Medium]]></description>
        <link>https://medium.com/@sauravomar01?source=rss-f4b7fddbf2d1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*R0Z7wpJdM0QURM7vVdOh8w.jpeg</url>
            <title>Stories by saurav omar on Medium</title>
            <link>https://medium.com/@sauravomar01?source=rss-f4b7fddbf2d1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:53:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sauravomar01/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Planning Is Not Execution (and Why Directors Shouldn’t Go Around Their EMs)]]></title>
            <link>https://sauravomar01.medium.com/planning-is-not-execution-and-why-directors-shouldnt-go-around-their-ems-22c9b0eab9ff?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/22c9b0eab9ff</guid>
            <category><![CDATA[engineering-leadership]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[leaders]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 15:36:41 GMT</pubDate>
            <atom:updated>2026-11-02T15:36:41.437Z</atom:updated>
            <content:encoded><![CDATA[<p>‘ve walked out of more than one planning review feeling really good. Roadmap agreed, dependencies sorted, dates that actually looked realistic for once. Everyone nodding.</p><p>And then a quarter later we’d delivered maybe half of it.</p><p>The annoying part was that the plan usually wasn’t the problem. When I looked back, it was the stuff that came after. Decisions we made in a meeting that nobody followed up on. A blocker that sat in someone’s standup update for two weeks. A change in priority that made total sense to me and that I never really explained to the team.</p><p>It took me a while to accept that planning and execution are different skills. I’d been treating them as one.</p><h3>Same manager, two very different jobs</h3><p>Planning is the fun part, if I’m honest. You get to step back and think. Where should the team be in six months? What are we going to say no to? Which bet actually matters? You can do most of it in a few good days with a whiteboard and a lot of chai.</p><p>Execution is nothing like that. It’s Tuesday afternoon, you have six things going on, and you notice a ticket hasn’t moved since last week. It’s the slightly awkward 1:1 where you ask why. It’s remembering to tell the team the date moved, and why it moved. Nobody gives you credit for any of it, and you have to keep doing it every week for months.</p><p>I know managers who write beautiful plans and get bored the moment the work starts. I also know managers who couldn’t write a strategy doc to save their life, but somehow always land things. Most of us lean one way. The useful thing is knowing which way you lean, and being a bit more deliberate about the other side.</p><h3>Where it actually falls apart</h3><p>It’s almost never one big failure. It’s small stuff that piles up.</p><p>The most common one I see is a decision with no owner. We all agree in the meeting, everyone leaves happy, and two weeks later the same topic comes back like we never talked about it. Nobody wrote down who was doing what next.</p><p>Then there are blockers. An engineer is waiting on another team, or on access, or on a design call. They mention it once in standup. After that it just sits there. If the manager’s head is already in next quarter’s planning, it can sit there for a long time.</p><p>Plans go stale too. Someone goes on leave, a dependency slips, something urgent shows up. That’s normal. What hurts is pretending the old dates still hold, because saying they don’t feels like admitting failure. By the time someone finally says it, there’s no room left to fix anything.</p><p>And the “why” fades. On day one the team knew exactly why this work mattered. By week six the priorities have shifted twice and nobody has said so out loud, so people are working hard on things that don’t matter as much anymore.</p><p>None of that gets solved in a planning doc. You solve it by showing up, every week, and paying attention.</p><h3>And then there’s the director who skips the EM</h3><p>There’s one more thing that breaks execution, and I’ve seen it often enough that it deserves its own section.</p><p>A director gets worried about a deadline. Or they just like being close to the work. So they start going straight to the engineers. A priority changes in a Slack DM. Someone gets asked to “just quickly pick this up” on a random call. The EM finds out a few days later, sometimes from their own report.</p><p>I understand why it happens. It feels faster. Why go through a layer when you can talk to the person actually doing the work? But the EM isn’t just a layer. They’re the one person holding the full picture for that team, and they’re the one who’ll be asked why things slipped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gkjj9EWM6G5p929oGxA3Tg.png" /></figure><p>On the left, direction comes from one place. On the right, the engineers are getting it from two, and the person responsible for delivery can only see half of it.</p><h3>What happens next</h3><p>I’ve watched this play out a few times, and it goes roughly the same way each time.</p><p>First, the EM starts losing respect. Nothing dramatic. The team just slowly figures out that the real decisions come from above. If your manager’s word can be overruled tomorrow, why take it too seriously today? The EM is still on the hook for delivery, but they’ve lost the authority to actually deliver.</p><p>The engineers end up with two bosses. One priority from the EM in the morning, a different one from the director after lunch. Most people will quietly follow whoever is more senior. Some try to do both and burn out. Either way, the EM’s plan stops meaning much.</p><p>The EM also loses track of what’s going on. Commitments get made in conversations they weren’t part of. They give leadership an update that’s already out of date. It’s embarrassing, and after a while some EMs just stop trying to manage dependencies at all.</p><p>Then things slip. And guess who usually gets blamed for it.</p><p>The director almost always means well, which makes it harder to call out. But every shortcut around the EM makes the team a little worse at running on its own.</p><h3>Skip-levels are fine. Going around the EM isn’t.</h3><p>To be clear, I’m not against directors talking to engineers. Good skip-levels are one of the best ways for a director to hear what’s really going on. People will tell a skip-level things they’d never put in a status report.</p><p>For me, the line is between listening and directing.</p><p>Ask engineers what’s slowing them down. Ask what they think of the roadmap. Take it all in. But when you hear something that needs to change, take it to the EM. Let them decide how to handle it, and let them be the one who tells the team. You can push the EM hard on the outcome, that’s your job. Just don’t go around them with the instruction.</p><p>If the real issue is that you don’t trust the EM to execute, that’s a different conversation, and it should happen openly, with the EM. Going around them doesn’t fix it. It only hides it for a while.</p><p>A few habits I’ve seen work well. Tell the EM before you do skip-levels, and share what you heard afterwards. When an engineer asks you to make a call, gently send them back to their EM. If a priority really has to change, change it with the EM in the room. And back them in public, even when you argue with them in private.</p><h3>One last thought</h3><p>A plan gets you started. Whether it works depends on the dull, steady stuff that comes after: following up, unblocking, adjusting, explaining. Week after week.</p><p>That work needs one owner who sits close to the team, and for most engineering teams that’s the EM. When the people above them respect that, teams get things done. When they keep reaching around, even a great plan falls apart, and everyone’s a bit confused about why.</p><p>If you’re a director, think about the last time you gave an engineer direction without their EM in the loop. Did it actually save time? Or did it just move the problem somewhere you couldn’t see it?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22c9b0eab9ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your Best Engineer Might Be Your Worst Manager Hire]]></title>
            <link>https://sauravomar01.medium.com/why-your-best-engineer-might-be-your-worst-manager-hire-eb1e96a80616?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb1e96a80616</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[leadership-skills]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 05:24:33 GMT</pubDate>
            <atom:updated>2026-10-31T05:24:33.722Z</atom:updated>
            <content:encoded><![CDATA[<p>Every team has that one engineer. The one who fixes the scary bugs, knows the whole system in their head, and ships twice as much as anyone else. When the manager role opens up, everyone already knows who’s getting it.</p><p>Six months later, something feels off. The new manager is in back-to-back meetings all day and still pushing code at midnight. PRs wait days for their review. One-on-ones keep getting cancelled. The team, which used to be fast, is now oddly slow.</p><p>And the engineer who used to love their job looks tired and a bit lost.</p><p>You didn’t just get a struggling manager. You also lost your best engineer. That’s a pretty expensive trade.</p><h3>Why we keep doing it</h3><p>Honestly, it makes sense on the surface. This person is the best at the work, the team respects them, and they know the system better than anyone. Who better to lead it?</p><p>There’s also a quieter reason. In a lot of companies, management is the only real way up. If you want a bigger title or a better salary after a certain point, you have to start managing people. So the promotion becomes a reward for being a great engineer, not a decision about who would be a great manager.</p><p>This isn’t just a tech thing. A <a href="https://nber.org/digest/may18/w24343.shtml">study by Benson, Li and Shue</a> looked at sales teams across 214 firms. The best salespeople were the most likely to get promoted into management. But on average, the stronger a new manager’s own sales record, the worse their team performed after the promotion.</p><p>That’s the Peter Principle in real data: people get promoted based on how well they did their last job, not on whether they’re suited for the next one.</p><h3>It’s a completely different job</h3><p>The biggest problem is that we call it a promotion, like it’s the same job but bigger. It isn’t. Almost everything about how you measure a good day changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pKIwXadCQGXGVYigeW2JSg.png" /></figure><p>The skills that made someone a great engineer, like going deep, owning the hard problem and being the person everyone asks, are often the exact habits that get in the way as a manager. Nobody tells them that on day one, so they keep doing what made them successful, and it slowly stops working.</p><h3>Signs it’s going wrong</h3><p>It rarely blows up all at once. It shows up in small, familiar ways.</p><p>The new manager still picks up the hardest tickets, because they’re faster than anyone else. Which is true, and also the problem. The team never gets the chance to grow into that work.</p><p>Code reviews turn into rewrites. Instead of explaining what could be better, they just fix it themselves. The engineer feels corrected, not coached, and stops trying the hard stuff.</p><p>They become the bottleneck. Every decision, every design, every release has to go through them. When they take a week off, the team more or less stops.</p><p>The people side gets pushed to the edges. One-on-ones get moved for “urgent” work. Feedback is either missing or comes out too blunt, because in their head the bar is their own level.</p><p>And they burn out. They’re doing a manager’s job during the day and their old job at night, and they’re not great at either anymore. Quite often they leave, and you lose both.</p><h3>Two types I’ve seen again and again</h3><p>Over the years, two patterns keep showing up for me.</p><h3>The brilliant jerk</h3><p>I’ve seen many examples of people who are really smart but just don’t listen. They’re usually right about the technical stuff, and they know it. But if you question their approach in a meeting, they get loud. Sometimes they literally shout.</p><p>After a few of those meetings, people stop disagreeing. Not because they’re convinced, but because it’s not worth the fight. Good ideas stay quiet, and the team slowly starts working around this person instead of with them. Eventually nobody wants to be on their team at all.</p><p>However smart they are, they’re not doing the team any good. They’re spoiling the culture, and that cost is bigger than whatever they ship.</p><h3>The person who is everywhere</h3><p>The second type is much nicer to work with, which is why it’s harder to spot. These are sensible, helpful people who try to solve problems for three or four teams at the same time. Every escalation comes to them. Every tricky decision waits for their opinion.</p><p>It feels great at first. But it causes two big problems.</p><p>First, nobody else grows. If one person always jumps in to solve the hard thing, the rest of the team never gets the chance to learn it.</p><p>Second, they become the bottleneck. Too many teams depend on them, so work piles up waiting for their time. When they’re on leave or stuck in another fire, everything slows down.</p><p>They’re everywhere, and because of that they’re nowhere. Spread across everything, they can’t go deep on anything. And that has a huge cost that rarely shows up in any review.</p><h3>What to do instead</h3><p>The fix isn’t “never promote great engineers.” Some of them turn into brilliant managers. The fix is to stop treating management as the default reward.</p><p>First, build a real path for engineers who want to stay engineers. Staff and principal roles should come with real money, real scope and real respect, not just a new title. If the only way to get a raise is to manage people, you’ll keep getting managers who never wanted the job.</p><p>Ask them if they actually want it. It sounds obvious, but a lot of people get promoted without anyone asking. And ask the deeper version too: “Do you want to spend most of your week on people problems instead of technical ones?”</p><p>Let them try it before it’s official. Give them a tech lead role, a small project with two or three people, or have them run hiring for a quarter. They’ll find out quickly whether they enjoy it, and so will you.</p><p>Make it okay to go back. If someone tries management and realises it’s not for them, moving back to engineering should be normal, not seen as failing. Otherwise people stay stuck in a job they don’t like because going back feels embarrassing.</p><p>And when someone does move, support them properly. Don’t just change the title and walk away. Give them a mentor, some training, and a manager who checks in on how the change is going, not just on delivery.</p><h3>If you’re the one being offered the role</h3><p>If you’re a strong engineer and someone just offered you a manager role, it’s flattering. It’s also okay to slow down before saying yes.</p><p>A few questions worth being honest with yourself about:</p><ul><li>When a teammate gets unblocked because of something you did, does that feel as good as shipping something yourself?</li><li>Are you okay with most of your day being meetings, conversations and context switching?</li><li>Can you watch someone solve a problem slower than you would, without taking over?</li><li>Are you saying yes because you want the job, or because it’s the only way to grow here?</li></ul><p>If the last one is the real reason, say that out loud to your manager. It might lead to a better conversation about the IC path than either of you expected.</p><h3>To wrap up</h3><p>Being a great engineer and being a great manager are both valuable. They’re just different jobs, and being good at one doesn’t mean you’ll be good at, or even enjoy, the other.</p><p>The companies that get this right stop asking “who’s our best engineer?” when a manager role opens up. They ask “who on this team already makes the people around them better?” Sometimes that’s the star engineer. Often it isn’t.</p><p>So look at your own team. If your best engineer got promoted tomorrow, would it be because they want to lead, or because there was nowhere else for them to go?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb1e96a80616" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Psychological Safety Isn’t About Being Nice]]></title>
            <link>https://sauravomar01.medium.com/psychological-safety-isnt-about-being-nice-258902c275db?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/258902c275db</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[leadership-development]]></category>
            <category><![CDATA[management-and-leadership]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 04:50:44 GMT</pubDate>
            <atom:updated>2026-10-31T04:50:44.849Z</atom:updated>
            <content:encoded><![CDATA[<p>I want to talk about a phrase that gets used a lot in leadership circles right now: psychological safety. I think a lot of us get it a bit wrong.</p><p>You’ve probably seen a team like this. Everyone is friendly. Standups start with weekend plans, nobody raises their voice, and every idea gets a “nice, good point.” Retros feel good.</p><p>Then the project slips by months. When you finally dig into what happened, you find out a couple of people saw the problem coming weeks ago. They just didn’t say anything.</p><p>Not because anyone would shout at them. They kept quiet because saying it would have spoiled the mood. Somewhere along the way, being nice had become more important than being honest.</p><p>That team felt safe. It wasn’t.</p><h3>So why do we mix up “safe” and “nice”?</h3><p>Part of it is just the word. “Safety” sounds soft. So when a leader hears “build psychological safety,” it easily turns into “be warm, avoid conflict, don’t upset anyone.”</p><p>Part of it is culture. In a lot of workplaces in India, we grow up respecting seniority and keeping the peace. Disagreeing with your manager in a meeting can feel rude. So people nod, and the manager takes the nodding as agreement.</p><p>What you end up with is a team where everyone is pleasant and nobody is really being straight with each other.</p><h3>What it actually means</h3><p>The term comes from Amy Edmondson at Harvard. Her definition is pretty simple: people on a team believe it’s safe to take interpersonal risks with each other.</p><p>By interpersonal risks, she means everyday stuff. Asking a question that might sound silly. Saying “I messed this up.” Telling someone senior you don’t think the plan will work. Sharing an idea before it’s fully thought through.</p><p>Google made the idea popular with Project Aristotle, a study of what makes their teams work well. Psychological safety came out as the biggest factor.</p><p>Nowhere in that definition does it say anything about comfort or everyone agreeing. It’s about whether you can tell the truth without paying for it later.</p><h3>Nice vs. safe</h3><p>From the outside, you honestly can’t tell the two apart. The difference only shows up when something goes wrong.</p><p>On a “nice” team, when a plan has a flaw, people nod in the meeting and fix it quietly later, or never. When the manager is wrong, everyone agrees anyway. Mistakes get played down. Feedback is so cushioned that nobody actually acts on it.</p><p>On a safe team, someone just says “I think there’s a problem here” in the meeting itself. People push back on the manager, politely, but they do it. Mistakes come out early so others can learn from them. Feedback is kind, but it’s clear.</p><p>What’s strange is that too much niceness can make a team less safe. If the unwritten rule is “don’t rock the boat,” the person who disagrees becomes the one rocking it. Speaking up turns into a risk, which is exactly what we were trying to get rid of.</p><p>Safe teams can be warm, of course. The warmth just doesn’t replace honesty.</p><h3>Safety alone isn’t enough</h3><p>Edmondson also talks about a second dial, which is accountability, or basically how high the bar is. Put the two together and you get four very different kinds of teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjPRyVahJzxC3IY-5LfJnA.png" /></figure><p>Real learning only happens in the top right, where people feel safe and the bar is still high. Most “nice” teams think they’re sitting in the comfort zone. But if being honest feels risky, they’re a lot closer to the anxiety zone than they realise.</p><p>So lowering the bar to make people comfortable doesn’t build safety. You just end up in the comfort zone, where nobody really grows.</p><h3>Safety also means someone has your back</h3><p>There’s one more part of this that I feel people don’t think about enough. Psychological safety also means knowing that if something goes wrong, your manager is going to back you, not throw you under the bus.</p><p>When people know one mistake won’t end their career, they start experimenting. They try new things and take sensible risks. Some of it won’t work. But some of it can bring really surprising results, and the team gets more confident along the way.</p><p>It does a lot for motivation too. If a team is scared, that fear is always going to show up in their productivity. You can’t do your best work while you’re busy protecting yourself.</p><p>I’ve seen many leaders directly call out the person who made a mistake, sometimes in front of everyone. I understand why it might feel like accountability. But honestly, they aren’t doing the company any good. They’re building a worse culture, and over time it hurts the company’s growth.</p><h3>Focus on the problem, not the person</h3><p>I’ve worked at companies on both sides of this. What I’ve noticed is that teams that focus on the problem instead of blaming someone end up doing better. They learn and keep iterating.</p><p>Here’s an example from Gojek. A developer made a change where cents got treated as dollars, and it cost the company a good amount of money. Instead of looking for someone to blame, the team did an RCA (root cause analysis) to understand how this got through. Then they made the process more rigorous, so the chances of it happening again were much lower.</p><p>In companies where the person just gets fired, the problem doesn’t actually get solved. The person is gone, but the gap in the process is still there. Eventually the same problem comes back, just with someone else.</p><h3>What you can actually do about it</h3><p>None of this needs an offsite or a big culture program. It’s mostly small things you keep doing again and again.</p><p>Talk about your own mistakes first. When the leader admits they got something wrong, everyone else feels allowed to do the same.</p><p>Stop ending meetings with “Any questions?” Ask “What am I missing?” instead. The first one usually gets silence. The second one gets real answers.</p><p>When someone brings you bad news, thank them before anything else. How you react in that first minute decides whether they’ll ever bring you bad news again.</p><p>Be tough on the work, not on the person. “This analysis has a gap” and “you missed something” land very differently.</p><p>And keep the bar clear. Safety doesn’t mean lower standards. Tell people what good looks like, and make it okay for them to say when they’re not there yet.</p><h3>To wrap up</h3><p>Think back to that friendly team from the start. They didn’t lack kindness. What they lacked was permission to be honest, and the confidence that someone would back them if things went wrong.</p><p>So here’s a question worth asking yourself: when was the last time someone on your team told you, straight up, that you were wrong?</p><p>If you can’t remember, your team is probably nice. I’m just not sure it’s safe.</p><p>I’d love to hear how you deal with this in your own teams.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=258902c275db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Redis Handles Thousands of Connections on One Thread]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sauravomar01.medium.com/how-redis-handles-thousands-of-connections-on-one-thread-6eb54bd7de8c?source=rss-f4b7fddbf2d1------2"><img src="https://cdn-images-1.medium.com/max/1600/1*OrzBCLknKDXXHxixKRN0nQ.png" width="1600"></a></p><p class="medium-feed-snippet">In my last post I went through how Redis Pub/Sub works under the hood, using the Google account linking flow we built at Gojek as the&#x2026;</p><p class="medium-feed-link"><a href="https://sauravomar01.medium.com/how-redis-handles-thousands-of-connections-on-one-thread-6eb54bd7de8c?source=rss-f4b7fddbf2d1------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sauravomar01.medium.com/how-redis-handles-thousands-of-connections-on-one-thread-6eb54bd7de8c?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6eb54bd7de8c</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[redis-cluster]]></category>
            <category><![CDATA[event-loop]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 09:31:01 GMT</pubDate>
            <atom:updated>2026-10-18T09:31:01.476Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How Redis Pub/Sub Actually Works Under the Hood]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sauravomar01.medium.com/how-redis-pub-sub-actually-works-under-the-hood-de7794ab2e40?source=rss-f4b7fddbf2d1------2"><img src="https://cdn-images-1.medium.com/max/1600/1*NJ5_S-zufNJkw1jOcyyM6g.png" width="1600"></a></p><p class="medium-feed-snippet">A while back at Gojek, I worked on the Google account linking flow. From the user&#x2019;s side it was simple: link your Google account once, and&#x2026;</p><p class="medium-feed-link"><a href="https://sauravomar01.medium.com/how-redis-pub-sub-actually-works-under-the-hood-de7794ab2e40?source=rss-f4b7fddbf2d1------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sauravomar01.medium.com/how-redis-pub-sub-actually-works-under-the-hood-de7794ab2e40?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/de7794ab2e40</guid>
            <category><![CDATA[redis-cluster]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[pub-sub]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 10:12:59 GMT</pubDate>
            <atom:updated>2026-10-17T10:12:59.284Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[LSM- Write Heavy Databases:]]></title>
            <link>https://sauravomar01.medium.com/lsm-write-heavy-databases-49e78b454da1?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/49e78b454da1</guid>
            <category><![CDATA[bloom-filter]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[lsm-tree]]></category>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 12:54:55 GMT</pubDate>
            <atom:updated>2021-01-08T12:54:55.656Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*CnjSaJmCEiLyAWQi.png" /></figure><h3>Introduction:</h3><p>Let’s think about what is the fasted way to write large datasets?. As we know the problem of disks being slow for random operations, but fast when accessed sequentially. If we append in a file or similar to how logging works. This idea of storing data into append-only logs is the basic idea behind LSM trees.</p><p>A <strong>log-structured merge-tree (LSM tree)</strong> is a data structure typically used when dealing with a write-heavy workload.</p><p>Log-Structured Merge is an important technique used in many modern data stores like BigTable, Cassandra, HBase, Riak, MongoDB 3.0(Wired Tiger).</p><h3>What is an LSM Tree?</h3><p>An LSM tree comprises of two or more levels of tree-like data structures. The simplest versions contain of two levels C0 and C1.</p><ul><li>first level is called the memtable and resides completely in-memory.</li><li>Second level is called as SStables it&#39;s large in size and is stored in a disk.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/0*okTdzYIK0hu_wfNx.png" /></figure><h3>First level or Memtable:</h3><ul><li>The underlying data structure is some form of a sorted tree like a <a href="https://en.wikipedia.org/wiki/Red%E2%80%93black_tree">red-black tree</a>.</li><li>So whenever writes come in, the data is added to this tree.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/0*aGK54ZqVry72Y5uI" /></figure><ul><li>Once writes get stored in this red-black tree until the tree reaches a its threshold thenit is flushed to disk as a segment on disk in sorted order which is know SStable</li><li>This allows us to write the segment file as a single sequential write even though the inserts may occur in any order.</li></ul><h3>Second level or SStable:</h3><ul><li>Once Memtable start growing and reaches its threshold then these sorted keys are written into disk in same order known as SSTables or “String Sorted Tables” .</li><li>SSTables are used to store key-value pairs in immutable log segments in the order sorted by string.</li><li>Lookups can be very efficient even if full hash index is not present.</li><li>Range queries are possible</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/0*5-zFUAym4JitApPe" /></figure><h3>How reading works?</h3><p>As of now we have seen and focus only on write path, let’s see how reads works</p><ul><li>In case of reading, given key is first looked up in the memtable.</li><li>In case of key is not present in Memtable, It uses sparse index (in memory index in elastic search it used to call it as inverted index).</li><li>Key is sorted and key is binary search to fetch the segments depending upon the status of the compaction.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cu39g4Avo4P1yWWS.png" /></figure><ul><li>In case of key is not present it still need to scan all segments.</li><li>To speed up in case of key is not present it uses <a href="https://medium.com/@sauravomar01/bloom-filter-probabilistic-space-efficient-data-structure-caaa2c09d7e0">bloom filter. </a>where it check if the key is not present no need to scan all segments.</li></ul><h3>Advantages of LSM tree</h3><ul><li>LSM Trees can handle very high write throughput</li><li>LSM Trees can be compressed better and thus result into smaller log segment files.</li></ul><h3>Disadvantages of LSM Tree</h3><ol><li>Compaction process sometime interfere with the performance of ongoing reads and writes.</li><li>Although use of bloom filter some how increase the performance in case of key not present, but if key exist then each key may exist at multiple places and thus checking if a key doesn’t exist needs all segments to be scanned.</li></ol><p>That’s It</p><p>Happy learning</p><p>Images which are used are copied from google, used just for reference not personal benefit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49e78b454da1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Re-Index Billions of Documents in Elastic-Search from different Cluster:]]></title>
            <link>https://sauravomar01.medium.com/re-index-billions-of-documents-in-elastic-search-from-different-cluster-41bc89ffe24b?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/41bc89ffe24b</guid>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[elasticsearch-reindex]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[reindex]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Thu, 06 Aug 2020 11:37:32 GMT</pubDate>
            <atom:updated>2020-08-06T11:37:32.597Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*XBWQt93J9g4__9Wp.png" /></figure><ul><li>We can use the reindex API to migrate indices data from one cluster to another cluster.</li><li>Elasticsearch provides backward compatibility support that enables indices from the previous major version to be upgraded to the current major version</li></ul><h4>Whitelist old cluster Node:</h4><ul><li>Setup a new cluster.</li><li>Added below property in one of the co-ordinators node.</li></ul><pre>reindex.remote.whitelist: old_cluster_node:9200</pre><ul><li>Above property whitelist <strong>old_cluster_node IP </strong>to reindex data from old cluster to new.</li><li>Set above property in <strong>elasticsearch.yml.</strong></li><li>And restart the node where you have added above property.</li><li>This property does not need to be added to all nodes of the cluster, only node where you are going to hit the Reindex API.</li></ul><h3>Re-Index Data:</h3><ul><li>Create an index of the appropriate mappings and settings.</li><li><strong>Set the </strong><strong>refresh_interval to </strong><strong>-1 and set </strong><strong>number_of_replicas to </strong><strong>0 for faster reindexing.</strong></li><li>Refresh Interval stops data to index and the number of replicas will stop replication and shard allocations which take huge time.</li><li>We can see the from the below example</li></ul><pre>curl --location --request PUT &#39;${NEW_CLUSTER_NODE_IP}/$I{INDEX_NAME}/_settings&#39; \</pre><pre>--header &#39;Content-Type: application/json&#39; \</pre><pre>--data-raw &#39;{</pre><pre>               {</pre><pre>                   &quot;index&quot; : {</pre><pre>                     &quot;refresh_interval&quot; : -1,</pre><pre>                      &quot;number_of_replicas&quot; : 0</pre><pre>                   }</pre><pre>               }</pre><pre>       }&#39;</pre><ul><li>Now we use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html"><strong>reindex</strong></a><strong> API</strong> to start re-indexing</li><li>If you run the reindex job in the background by setting wait_for_completion to false, the reindex request returns a task_id you can use to monitor the progress of the reindex job with the<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/tasks.html"> task API:</a> GET _tasks/TASK_ID.</li></ul><pre>curl --location --request PUT &#39;${NEW_CLUSTER_NODE_IP}/_reindex?wait_for_completion=false&#39; \</pre><pre>--header &#39;Content-Type: application/json&#39; \</pre><pre>--data-raw &#39;{</pre><pre>          &quot;source&quot;: {</pre><pre>             &quot;remote&quot;: {</pre><pre>                   &quot;host&quot;: &quot;${PRIMARY_SERVER_URL}&quot;,</pre><pre>                   &quot;socket_timeout&quot;: &quot;5m&quot;,</pre><pre>                   &quot;connect_timeout&quot;: &quot;5m&quot;</pre><pre>               },</pre><pre>           &quot;index&quot;: &quot;${OLD_INDEX_NAME}&quot;</pre><pre>        },</pre><pre>         &quot;dest&quot;: {</pre><pre>              &quot;index&quot;: &quot;${NEW_INDEX_NAME}&quot;</pre><pre>       }</pre><pre>   }&#39;</pre><p><strong>Parameter Explanations:</strong></p><p><strong>host</strong>: The REST endpoint of the remote cluster.</p><p><strong>socket_timeout</strong>: Time UnitThe wait time for socket reads (default 30s).</p><p><strong>connect_timeout</strong>: Time UnitThe wait time for remote connection timeouts (default 30s)</p><ul><li>After reindexing completed:</li></ul><h4>Set the refresh_interval to 10s and set number_of_replicas to 2</h4><pre>curl --location --request PUT &#39;${NEW_CLUSTER_NODE_IP}/$I{INDEX_NAME}/_settings&#39; \</pre><pre>--header &#39;Content-Type: application/json&#39; \</pre><pre>--data-raw &#39;{</pre><pre>{</pre><pre>         &quot;index&quot; : {</pre><pre>           &quot;refresh_interval&quot; : 10s,</pre><pre>           &quot;number_of_replicas&quot; : 3</pre><pre>        }</pre><pre>      }</pre><pre>}&#39;</pre><p>Happy Learning!!</p><p>Thanks</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41bc89ffe24b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wide Vs Narrow dependencies in Apache Spark:]]></title>
            <link>https://sauravomar01.medium.com/wide-vs-narrow-dependencies-in-apache-spark-2cd33bf7ed7d?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cd33bf7ed7d</guid>
            <category><![CDATA[rdd]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark-rdd]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 15:48:32 GMT</pubDate>
            <atom:updated>2020-06-27T15:56:55.894Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/312/0*eIp0PhPXsTM7oQh4" /><figcaption>#Copied from Google</figcaption></figure><h3>What is Spark?</h3><p>Wikipedia says “ <em>Apache Spark is an open-source distributed general-purpose cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault toleranc</em>e.”</p><ul><li>Apache Spark is a lightning-fast unified analytics engine for big data and machine learning.</li><li>It was originally developed at UC Berkeley in 2009.</li><li>Companies like Netflix, Yahoo, and eBay have deployed Spark at a massive scale, collectively processing multiple petabytes of data on clusters of over 8,000+ nodes.</li><li>It is the largest open source community in big data, with over 1000 contributors from 250+ organizations.</li><li>If you have large amounts of data that require low latency processing that a typical <strong>MapReduce </strong>program cannot provide, Spark is the way to go.</li><li>It speeds up to 100 times faster than Map Reduce for iterative algorithms or interactive data mining.</li><li>Computations are represented in Spark as a DAG(<em>Directed Acyclic Graph</em>)officially described as a <em>lineage graph </em>over RDDs,</li><li><em>Lineage Graph</em> represents data distributed across different nodes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*es9fFBQPEx1TjXNO" /><figcaption>#Copied from Google</figcaption></figure><h3>Transformation:</h3><p>Transformations is a kind of process that will transform your RDD data from one form to another in Spark. and when you apply this operation on an RDD, you will get a new RDD with transformed data (RDDs in Spark are immutable).</p><ul><li>Transformations are the core of how you will be expressing your business logic using DAG in Spark and intermediate operation conversion from one form to another RDD.</li></ul><h4>Two types of transformations in SPARK:</h4><ul><li>Wide Transformations</li><li>Narrow Transformations</li></ul><p>Let’s see one by one.</p><h3>Narrow Transformations:</h3><p>These types of transformations convert each input partition to only one output partition. When each partition at the parent RDD is used by at most one partition of the child RDD or when each partition from child produced or dependent on single parent RDD.</p><ul><li>This kind of transformation is basically fast.</li><li>Does not require any data shuffling over the cluster network or no data movement.</li><li>Operation of map()and filter() belongs to this transformations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/0*we4HwhTVbuKJxBRi" /><figcaption>#Copied from Google</figcaption></figure><h3>Wide Transformations:</h3><p>This type of transformation will have input partitions contributing to many output partitions. When each partition at the parent RDD is used by multiple partitions of the child RDD or when each partition from child produced or dependent on multiple parent RDD.</p><ul><li>Slow as compare to narrow dependencies speed might be significantly affected as it might be required to shuffle data around different nodes when creating new partitions.</li><li>Might Require data shuffling over the cluster network or no data movement.</li><li>Functions such as groupByKey(), aggregateByKey(), aggregate(), join(), repartition() are some examples of wider transformations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/0*TK6fj3i_gD7J-jtS" /></figure><p>When working with Spark, it is always good or keeps in mind all operations or transformations Which might require data shuffling and hence slow down the process. Try to optimize and reduce the usage of wide dependencies as much as you can.</p><p>Happy Learning!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cd33bf7ed7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2PC: Two-phase commit protocol]]></title>
            <link>https://sauravomar01.medium.com/2pc-two-phase-commit-protocol-4ba47e441cdf?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ba47e441cdf</guid>
            <category><![CDATA[distributed-transaction]]></category>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[acid]]></category>
            <category><![CDATA[2pc]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Mon, 22 Jun 2020 14:22:31 GMT</pubDate>
            <atom:updated>2020-06-22T14:22:31.835Z</atom:updated>
            <content:encoded><![CDATA[<p>In a distributed system, transactions involve altering data on multiple databases, Which causes the processing to be more complicated since the database has to coordinate with the other operable units for committing or rolling back of changes in a transaction as a self-contained unit. Either the entire transaction commits or the entire transaction rolls back. When the data is spread across different DC’s in distributed computing, each server is an independent entity with separate log records, the process can become more complex.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*B5DzmQAT0zTXXe-x.png" /></figure><ul><li>The two-phase commit protocol (2PC) has been used in different enterprise software for a couple of years.</li><li>Oracle, IBM DB2, PostgreSQL, MariaDB, VoltDB, Cloud Spanner, Apache Flink, Apache Kafka, and Azure SQL Database uses this.</li><li>If your system supports ACID transactions across shards/partitions/databases, there’s a high probability that it is running 2PC.</li></ul><h3>How does 2PC commit works?</h3><p>This protocol has two phases that is why it is called as Two-phase commit protocol.</p><ul><li>The first phase is called as <strong>PREPARE</strong></li><li>The second phase is called as <strong>COMMIT</strong>.</li></ul><p>Two units were used Co-ordinator and Participator(or worker).</p><ul><li>Co-ordinator: who initiate the transactions</li><li>Participator: which participates in these transactions.</li></ul><p><em>Not clear don’t worry you will understand in upcoming descriptions</em></p><p><strong>Explanation</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/0*VeZfJ2mLz5UNYv89.png" /></figure><p><strong>Phase1</strong>: The co-ordinator asks from each participator whether they have successfully completed their responsibilities for that transaction and are ready to commit. Each participator responds ‘yes |OK’ or ‘no|abort’.</p><ul><li>Every participator writes its data records in a log. If it is unsuccessful to do, then it responds with a failure message; if it is successful, then it sends an OK message.</li></ul><p><strong>Phase2</strong>: This phase will start when Co-ordinator receive successful response in phase 1 from all participator.</p><ul><li>Now Co-ordinator sends a commit request to all the participators.</li><li>Now participators write the commit as part of its log record.</li><li>Participants send a message that its commit has been successfully implemented.</li><li>If a server fails, the coordinator sends instructions to all servers to roll back the transaction</li></ul><p>This mechanism ensures the ACID transactions guarantee</p><p><strong>Atomicity</strong>: either the entire transaction will be reflected in the final state of the system or none of it.</p><p><strong>Consistency</strong>: Either Transaction is successful or its rollback.</p><p><strong>Isolation</strong>: Concurrent transactions do not interfere with each other</p><p><strong>Durability</strong>: After successful completion of a transaction all changes made by the transaction persist even in the case of a system failure.</p><p>Thanks</p><p>Happy Learning!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ba47e441cdf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Disable SELinux on Ubuntu:]]></title>
            <link>https://sauravomar01.medium.com/disable-selinux-on-ubuntu-75f1c1f6592b?source=rss-f4b7fddbf2d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/75f1c1f6592b</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[selinux]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[apparmor]]></category>
            <dc:creator><![CDATA[saurav omar]]></dc:creator>
            <pubDate>Fri, 19 Jun 2020 11:23:08 GMT</pubDate>
            <atom:updated>2020-06-19T11:24:10.749Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><ul><li>Also known as Security Enhanced Linux.</li><li><a href="https://en.wikipedia.org/wiki/Security-Enhanced_Linux">SELinux</a> is a MAC (<a href="https://en.wikipedia.org/wiki/Mandatory_access_control">mandatory access control</a>)module present in the kernel of Linux systems. It is developed by <a href="https://en.wikipedia.org/wiki/Red_Hat">Redhat</a> and <a href="https://en.wikipedia.org/wiki/National_Security_Agency">NSA</a> and released around 1998.</li><li>It is an additional layer of security services on top of the standard Linux DAC (Discretionary access control) User/Group based permissions.</li><li>It adds protection to your Linux operating system by not allowing everything that has not been specifically allowed.</li><li>It is an open-source application that protects the host by isolating each application and confining its activities.</li><li>By default, Ubuntu uses <a href="https://wiki.ubuntu.com/AppArmor">AppArmor</a> and not SELinux,</li></ul><h4><strong>Disable SELinux on Ubuntu:</strong></h4><ul><li>By default, it is installed in Ubuntu and you can validate by running the following command:</li></ul><pre>sestatus</pre><pre><strong>SELinux status:                 enabled<br></strong>SELinuxfs mount:                /sys/fs/selinux<br>SELinux root directory:         /etc/selinux<br>Loaded policy name:             targeted<br>Current mode:                   enforcing<br>Mode from config file:          enforcing<br>Policy MLS status:              enabled<br>Policy deny_unknown status:     allowed<br>Max kernel policy version:      28</pre><p>As we can see SELinux is enabled.</p><h4>Disable SELinux Temporarily:</h4><ul><li>The following methods only works until the system does not reboot.</li><li>It will update for the current shell.</li></ul><pre># echo 0 &gt; /selinux/enforce</pre><pre>        or </pre><pre># setenforce 0</pre><pre><br>        or </pre><pre># setenforce Permissive</pre><h4>Disable SELinux Permanently:</h4><p>To permanently disable SELinux, use any text editor to open the file /etc/sysconfig/selinux as follows and set:</p><pre>SELINUX=disabled</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*ViTLaizl0jmDu1LX.png" /></figure><ul><li>Update and close the file, and don’t forget reboot then only it will start working.</li><li>After Reboot.</li><li>Validate, it should look like below.</li></ul><pre>$ sestatus</pre><pre><strong>SELinux status:                 disabled</strong><strong><br></strong>SELinuxfs mount:                /sys/fs/selinux<br>SELinux root directory:         /etc/selinux<br>Loaded policy name:             targeted<br>Current mode:                   enforcing<br>Mode from config file:          enforcing<br>Policy MLS status:              enabled<br>Policy deny_unknown status:     allowed<br>Max kernel policy version:      28</pre><p>Thanks</p><p>Happy Learning!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75f1c1f6592b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>