<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Deependra Raliya on Medium]]></title>
        <description><![CDATA[Stories by Deependra Raliya on Medium]]></description>
        <link>https://medium.com/@deependraraliya?source=rss-189b213197de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RUhdj8gih5y0u_SpDIo7DQ.png</url>
            <title>Stories by Deependra Raliya on Medium</title>
            <link>https://medium.com/@deependraraliya?source=rss-189b213197de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:26:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deependraraliya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why I Built My Own Google Colab?]]></title>
            <link>https://deependraraliya.medium.com/why-i-built-my-own-google-colab-2f1140cd5bf9?source=rss-189b213197de------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f1140cd5bf9</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Deependra Raliya]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 15:02:43 GMT</pubDate>
            <atom:updated>2026-09-29T15:02:43.945Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0Lo-zrb8Ce66BKWcGUP_w.jpeg" /></figure><p>In this post I’ll be sharing my experience about how I turned two bare-metal servers and four NVIDIA H200 GPUs into a self-service compute platform for a regulated environment.</p><p>There is a difference between having a lot of compute and actually having a compute platform.</p><p>Having four NVIDIA H200 GPUs, hundreds of CPU cores, terabytes of RAM, and tens of terabytes of storage sounds like the hard part is already solved. In reality, the hardware is only the beginning.</p><p>The real engineering problem is figuring out how to make that infrastructure useful to dozens of people at the same time, while keeping workloads isolated, resources controlled, GPUs efficiently utilized, and proprietary data inside the organization’s environment.</p><p>That was the problem I was given.</p><p>I was handed two powerful bare-metal servers and essentially told: we have researchers, developers, ML engineers, and AI engineers building everything from agentic AI systems to machine learning models for life sciences. Make this infrastructure useful and make sure we get as much value out of it as possible.</p><p>I was given a lot of freedom to decide how to build it. I would research different architectural approaches, propose them to our engineering manager or director, discuss the security, monitoring, auditability, reliability, and operational considerations, iterate on the design, and then implement it.</p><p>I am genuinely grateful for the amount of trust that was placed in me with this project. It gave me the opportunity to work on the problem end to end, from bare-metal infrastructure and Kubernetes all the way to the developer experience.</p><p><strong>And that led to a simple question:</strong></p><p>What if we could give everyone an experience similar to Google Colab, but running entirely on our own infrastructure?</p><h3>THE PROBLEM</h3><p>I work in the pharmaceutical domain, where a significant amount of the data and intellectual property we work with is proprietary.</p><p>That makes the infrastructure problem fundamentally different from simply spinning up a cloud GPU and starting a training job.</p><p>Our organization deliberately invested in substantial on-premise infrastructure because we have teams of PhDs, developers, ML engineers, and AI engineers working on problems such as agentic AI, model training, inference, and other life-science workloads.</p><p>The hardware I was given consisted of two identical bare-metal servers.</p><p><strong>Each server has:</strong></p><p>130 AMD CPU cores. Approximately 1 TB of RAM. Approximately 25 TB of storage. 2 NVIDIA H200 GPUs</p><p><strong>Across both servers, that gives us:</strong></p><p>260 AMD CPU cores, Approximately 2 TB of RAM, Approximately 50 TB of storage, 4 NVIDIA H200 GPUs</p><p>The hardware was powerful enough to support a serious amount of ML and AI work.</p><p><strong>But there was an obvious problem.</strong></p><p>How do you allow dozens of engineers and researchers to use that infrastructure without turning the servers into a shared playground where everyone can interfere with everyone else’s work?</p><p>How do you make sure an expensive H200 is not sitting idle for hours?</p><p>How do you make sure one workload cannot consume all the available memory?</p><p>How do you make sure production inference does not get affected by someone’s experimental training job?</p><p>How do you give people an environment where they can start working without spending hours installing Python, CUDA, PyTorch, Jupyter, and every other dependency they need, <strong>in an airgapped environment</strong>?</p><p>And perhaps most importantly, how do you do all of this while keeping the data and workloads entirely within the organization’s infrastructure?</p><p>That was the engineering problem I wanted to solve.</p><h3>THE “JUST GIVE EVERYONE SSH” APPROACH</h3><p>The simplest solution would have been to create Linux accounts and give everyone SSH access to the servers.</p><p>It would work.</p><p>A developer could SSH into a machine, create a virtual environment, install Python packages, download dependencies, and start working.</p><p>But once you have many people sharing the same machines, things get complicated very quickly.</p><p>Different workloads need different versions of Python, PyTorch, CUDA libraries, system packages, and other dependencies. One person’s environment can interfere with another person’s environment. Disk space can get consumed unexpectedly. Long-running processes can consume CPU and memory that someone else needs.</p><p>GPUs make this problem even more interesting.</p><p>Imagine three people are using the same GPU. One workload suddenly consumes far more GPU memory than expected and triggers an out-of-memory condition. Now you potentially have multiple workloads sharing the same physical GPU being affected by one person’s experiment.</p><p>The same problem exists with system memory, but the consequences can be even broader.</p><p>Someone could accidentally load a 2 TB dataset into a NumPy matrix entirely in memory. The resulting OOM could affect the entire server and potentially take down unrelated workloads, including production services.</p><p>And while nobody is going to intentionally run something destructive like:</p><pre>rm -rf /</pre><p>On a production server, the fact that it is technically possible is itself a reason to introduce boundaries.</p><p>The infrastructure should not depend entirely on everyone using it correctly all the time.</p><p>There is another limitation as well. Linux users are not a particularly convenient abstraction for dynamically allocating compute resources. You can build controls around individual users, but it becomes cumbersome to provide the kind of resource isolation and dynamic allocation that an ML platform needs.</p><p>I wanted to solve this at the infrastructure level.</p><p>Instead of trusting every user to behave correctly on a shared host, I wanted the platform to enforce the boundaries.</p><p>A user should get only the CPU, RAM, GPU, and storage that has been allocated to their environment.</p><p>And their environment should be isolated from everyone else’s.</p><h3>THE IDEA: MAKE THE ENVIRONMENT DISPOSABLE</h3><p>This led me to a much better model.</p><p>Instead of giving developers, researchers, and ML engineers direct access to the underlying servers, I wanted to give each person an isolated environment.</p><p>The environment would be created when they needed it and removed when they were finished.</p><p>Their data would remain persistent, but their compute environment could be disposable.</p><p>This distinction turned out to be extremely useful. The user would interact with the platform through a browser.</p><p>They would request the amount of compute they needed, and the platform would create an isolated Docker environment with those resources allocated to it.</p><p><strong>The experience would be something like this:</strong></p><p>Open a browser. Log into JupyterHub. Request compute. Get a Jupyter environment. Start working.</p><p>There is no SSH session to manage. There is no need to configure CUDA on a laptop. There is no need to install Jupyter. There is no need to manually find an available GPU.</p><p>The infrastructure becomes something that the user consumes rather than something they have to operate.</p><p>That was the experience I wanted to build. In other words, I wanted to build our own internal version of Google Colab.</p><h3>THE INFRASTRUCTURE</h3><p>Before getting into JupyterHub, it is important to understand the underlying infrastructure. I started with two identical bare-metal servers.</p><p><strong>The architecture is intentionally simple:</strong></p><pre>                  Kubernetes Cluster<br><br>              +---------------------------+<br>              |                           |<br>              |       Control Plane       |<br>              |                           |<br>              |        Server 1           |<br>              |                           |<br>              |      2 × NVIDIA H200      |<br>              |      130 AMD CPU cores    |<br>              |      ~1 TB RAM            |<br>              |      ~25 TB storage       |<br>              |                           |<br>              +-------------+-------------+<br>                            |<br>                            |<br>                            |<br>              +-------------v-------------+<br>              |                           |<br>              |          Worker           |<br>              |                           |<br>              |        Server 2           |<br>              |                           |<br>              |      2 × NVIDIA H200      |<br>              |      130 AMD CPU cores    |<br>              |      ~1 TB RAM            |<br>              |      ~25 TB storage       |<br>              |                           |<br>              +---------------------------+</pre><p>Both servers have the same hardware configuration.</p><p>Server 1 runs the Kubernetes control plane, while Server 2 is joined as a worker node.</p><p>I used Ansible to keep the software configuration consistent across both machines. Kubernetes was bootstrapped using kubeadm, and Calico was used as the CNI.</p><p>This entire setup was automated with Ansible, including the Kubernetes bootstrap process. You can find the ansible playbook I used to setup kubernetes <a href="https://github.com/DeependraRaliya/K8s-bootstrap-ansible">here</a>.</p><p>The idea behind Kubernetes here was not to make JupyterHub run on Kubernetes.</p><p>The idea was to create an abstraction over the two physical machines. Without Kubernetes, you effectively have two separate servers.</p><p>With Kubernetes, they become part of a single cluster that can expose compute resources as a unified infrastructure layer.</p><p>The servers still physically exist, of course, but from the perspective of the platform, I can reason about the cluster as a larger pool of resources.</p><p>That abstraction became particularly useful once GPUs entered the picture.</p><h3>HOW THE GPU LAYER FITS IN</h3><p>The next problem was making the NVIDIA H200 GPUs usable as managed infrastructure.</p><p>For that, I installed the NVIDIA GPU Operator and NVIDIA KAI Scheduler.</p><p>The GPU Operator provides the Kubernetes integration required to expose and manage NVIDIA GPUs.</p><p>KAI Scheduler provides GPU-aware scheduling capabilities and gives the cluster a much better understanding of GPU workloads.</p><p>One of the things I wanted to be able to do was partition GPU resources rather than treating every H200 as a single indivisible resource.</p><p>For the Jupyter workloads, I used NVIDIA MIG to create GPU slices and allocated those slices to individual environments.</p><p>This gives us a much more granular way of sharing expensive GPU hardware.</p><p><strong>Instead of thinking only in terms of:</strong></p><p>One user gets one entire H200.</p><p><strong>I can think in terms of:</strong></p><p>This user gets this GPU slice. Another user gets another slice. Another workload gets a different slice.</p><p>That makes the GPU pool significantly more useful for smaller workloads and experimentation.</p><p>I also built resource allocation around this.</p><p>The platform maintains a JSON-based allocation state containing information about which GPU resources have been assigned, who they are assigned to, how much other compute has been allocated, and which resources are currently idle.</p><p>This allowed me to build lifecycle management around the compute environments.</p><p>If a user’s GPU environment remains idle for some time, the allocated GPU resource is released and the spawned container is terminated.</p><p>The important part is that the user’s data is not deleted. Their notebooks, files, and persistent data remain intact.</p><p>When they need the environment again, they can request it and the platform can recreate it in roughly 10 to 30 seconds, with their previous data and notebooks available again.</p><p>This changes the economics of the platform considerably. A user does not need to permanently occupy an expensive GPU just because they might need it again later.</p><p>The compute can be ephemeral while the data remains persistent.</p><h3>KUBERNETES IS ALSO USED FOR INFERENCE</h3><p>The Kubernetes layer is not only there for GPU resource management. We also use it to run inference workloads.</p><p>For example, internally developed models and LLMs are served using vLLM, Ollama, TensorFlow on our own infrastructure.</p><p>This is particularly useful for workloads where we want to use internally developed or proprietary models without sending data to external AI services.</p><p><strong>The objective is simple:</strong></p><p>The data stays on-premise. The models stay on-premise. The inference workloads stay on-premise.</p><p>The infrastructure becomes the controlled boundary around the entire AI workflow.</p><p>This is particularly important in a pharmaceutical environment where proprietary information can be part of the data being processed.</p><h3>HOW JUPYTERHUB FITS INTO THE PICTURE</h3><p>Once the infrastructure and GPU layer were in place, I needed to solve the developer experience.</p><p>Kubernetes is excellent infrastructure, but it is not the interface I wanted developers and researchers to use.</p><p>I wanted them to see a simple browser-based interface. That is where JupyterHub came in.</p><p>The JupyterHub architecture is separate from the Kubernetes scheduling layer.</p><p><strong>The flow looks like this:</strong></p><pre>Browser<br>                        |<br>                        v<br>                   JupyterHub<br>                        |<br>                        v<br>                Resource Request<br>            CPU / RAM / GPU / Storage<br>                        |<br>                        v<br>                  DockerSpawner<br>                        |<br>                        v<br>             User-specific container<br>                        |<br>         +--------------+--------------+<br>         |                             |<br>         v                             v<br>   Persistent Storage          Read-only Shared Data<br>         |                             |<br>   User notebooks              Large datasets<br>   Experiment data             Shared models<br>   Checkpoints                 Common resources</pre><p>JupyterHub handles the user-facing experience, while DockerSpawner is responsible for creating the user’s Docker environment.</p><p>When someone requests an environment, the platform determines the resources they are allowed to use and spawns a container with those resources allocated.</p><p>The user gets their own isolated environment without having to interact directly with the host operating system.</p><p>This is where the platform starts to feel like Google Colab. A developer or ML engineer does not need to think about which server has an available GPU.</p><p>They do not need to SSH into a machine. They do not need to install the environment themselves. They simply request the resources and start working.</p><h3>RESOURCE ISOLATION</h3><p>Resource isolation was one of the main reasons for building the platform this way.</p><p>The whole point was not just convenience. It was safety.</p><p>If someone needs 8 CPU cores, 32 GB of RAM, and a GPU slice, they should get those resources rather than unrestricted access to the server.</p><p>The spawned Docker container provides the boundary around the workload.</p><p>This lets the platform enforce resource allocation at the container level rather than relying on every user to manually manage their resource consumption.</p><p>It also means that the platform can make resource allocation explicit.</p><p><strong>For example, an environment might be given:</strong></p><p>16 CPU cores, 64 GB RAM, One GPU slice, A specific amount of storage</p><p><strong>Another environment might need only:</strong></p><p>4 CPU cores, 16 GB RAM</p><p>The infrastructure can accommodate both without treating them as equivalent workloads.</p><p>This is particularly important when you have expensive GPUs and many users competing for them.</p><h3>STORAGE IS PART OF THE PLATFORM TOO</h3><p>Compute is only useful if users can work with data.</p><p>Each spawned environment therefore has access to persistent user-specific storage as well as shared data.</p><p>The user-specific storage is where notebooks, experiment outputs, checkpoints, and other personal data can live.</p><p>The shared storage contains large datasets, models, and other resources that need to be accessible to multiple users.</p><p>The shared data is mounted read-only. This was an important design choice.</p><p>If ten people need access to the same multi-terabyte dataset, I do not want ten copies of that dataset.</p><p>At the same time, I do not want someone accidentally modifying or deleting the source dataset while another team is using it.</p><p><strong>So the model is:</strong></p><p>User-specific storage is persistent and writable.</p><p>Shared datasets are accessible to everyone who needs them, but mounted read-only.</p><p>This gives users the flexibility they need while protecting common data from accidental modification.</p><h3>WHY NOT JUST USE GOOGLE COLAB?</h3><p>At this point, the obvious question is:</p><p>If Google Colab already provides browser-based notebooks and compute, why build something similar?</p><p>Because the problem was never simply “we need Jupyter notebooks.” The actual problem was much broader.</p><p>We needed to make our existing on-premise compute available to a large number of developers, researchers, and ML engineers in a way that was easy to use, resource-efficient, isolated, and compatible with the requirements of a regulated environment.</p><p>We already had the hardware. We had proprietary data. We had teams building ML models and AI applications.</p><p>What we needed was an abstraction that made all of that infrastructure easy to consume.</p><p>Google Colab solves a different version of that problem. What I wanted was the same simplicity of:</p><p>Open a browser. Get an environment. Start working.</p><p>But with the compute, data, models, and workloads remaining inside our own infrastructure.</p><h3>THE PRODUCTION AND RESEARCH SPLIT</h3><p>There is one more architectural decision that I consider particularly important.</p><p>Not all GPU workloads are equal.</p><p>A production inference workload should not have to compete directly with an experimental training job.</p><p>If someone starts a large training run and consumes all available GPU capacity, I do not want that to affect a production service.</p><p>So I divided the four H200 GPUs into two logical pools. Two GPUs are reserved for production inference.</p><p>The other two are available for training, research, experimentation, and development.</p><p><strong>The model looks like this:</strong></p><pre>4 × H200 GPUs<br>                       |<br>         +-------------+-------------+<br>         |                           |<br>         v                           v<br>  Production Pool              Research Pool<br>      2 GPUs                      2 GPUs<br>         |                           |<br>    Inference             Training / Research<br>                          Experiments / Dev</pre><p>This separation gives us a clear resource boundary.</p><p>Production workloads have dedicated capacity, while research workloads can use the remaining GPUs dynamically.</p><p>It also makes the platform easier to reason about operationally.</p><p>A production service does not suddenly become dependent on whether someone is running a 12-hour experimental training job.</p><h3>WHAT WE ENDED UP BUILDING</h3><p>After putting all of these pieces together, the platform became much more than a collection of infrastructure components.</p><p>At the infrastructure level, we have two identical bare-metal servers:</p><pre>                    KUBERNETES CLUSTER<br>                           │<br>              ┌────────────┴────────────┐<br>              │                         │<br>              ▼                         ▼<br>       ┌───────────────┐         ┌───────────────┐<br>       │   SERVER 1    │         │   SERVER 2    │<br>       │ Control Plane │         │    Worker     │<br>       ├───────────────┤         ├───────────────┤<br>       │ 130 AMD cores │         │ 130 AMD cores │<br>       │ ~1 TB RAM     │         │ ~1 TB RAM     │<br>       │ ~25 TB storage│         │ ~25 TB storage│<br>       │ 2 × H200      │         │ 2 × H200      │<br>       └───────────────┘         └───────────────┘</pre><p>Ansible keeps the software configuration consistent across the two servers.</p><p>Kubeadm bootstraps the Kubernetes cluster. Calico provides the cluster networking.</p><p>The NVIDIA GPU Operator integrates the H200 GPUs with Kubernetes. KAI Scheduler provides GPU-aware scheduling. MIG allows GPU resources to be partitioned for appropriate workloads.</p><p>Kubernetes also hosts inference workloads such as vLLM and Ollama, allowing internally developed models and LLMs to run within the organization’s infrastructure.</p><p>On top of that, JupyterHub provides the self-service developer experience.</p><p>DockerSpawner creates the user’s isolated environment with the resources assigned to them.</p><p>Persistent storage keeps their data intact even when their compute environment is released.</p><p>Shared datasets are mounted read-only so that everyone can use them without modifying the source data.</p><p>Idle environments can release expensive compute resources automatically, while users can recreate their environment later and continue where they left off.</p><p>From the user’s perspective, all of this complexity disappears.</p><p>They open a browser. They request compute. Their environment starts. They work.</p><p>When they are finished, the compute can eventually be released without deleting their data.</p><h3>THE BIGGER IDEA</h3><p>The most important thing I learned from this project is that building an ML platform is not really about putting together a collection of ML tools.</p><p>It is about creating the right abstraction between people and infrastructure.</p><p>A developer should not have to think about which physical server has an available GPU.</p><p>An ML engineer should not have to SSH into a machine just to start a training experiment.</p><p>A researcher should not have to spend half a day setting up CUDA and Python dependencies before they can run a notebook.</p><p>At the same time, the infrastructure team should not have to manually provision a machine every time someone needs compute.</p><p>The platform should sit between both sides. The user describes what they need. The platform handles how it is provided.</p><p>That is the fundamental idea behind what I built. The physical infrastructure becomes an implementation detail. The compute platform becomes the product.</p><h3>WHAT I LEARNED</h3><p>One of the biggest lessons from this project was that infrastructure becomes significantly more valuable when you stop exposing infrastructure itself and start exposing capabilities.</p><p>A user does not really care that there are two physical servers. They care that they can get a GPU. They do not care that Kubernetes is running underneath the platform.</p><p>They care that their environment starts reliably. They do not care which node their workload runs on. They care that they have enough CPU, memory, GPU, and storage to do their work.</p><p>And they definitely should not have to understand the infrastructure just to run a machine learning experiment.</p><p>That shift in perspective is what turned this project from “setting up Kubernetes” into building an actual internal ML platform.</p><p>It also changed how I think about infrastructure engineering in general. The goal is not always to expose more infrastructure.</p><p>Sometimes the best infrastructure is the infrastructure that users barely have to think about.</p><h3>ACKNOWLEDGEMENTS</h3><p>I want to take a moment to acknowledge the people who supported me throughout this project. I was given the ownership to research, design, architect, and implement this platform end to end, and I am genuinely grateful to my organization for trusting me with a project of this scale and giving me the freedom to take that ownership.</p><p>The architecture, implementation, automation, infrastructure setup, and platform integration were my responsibility, but building something for a highly regulated AI environment also meant thinking beyond just getting the technology to work. Our organization follows AI governance principles aligned with the NIST AI Risk Management Framework, and throughout the project I had valuable discussions with my Engineering Manager and other teams who challenged my architectural decisions and helped me account for industry standards, engineering best practices, security principles, regulatory requirements, auditability, monitoring, and operational safety. Their feedback helped me strengthen the design and consider perspectives that are easy to overlook when you are focused primarily on the technical implementation.</p><p>I am especially thankful for the trust and autonomy I was given to take a problem from an initial set of bare-metal resources to a working internal ML platform, while having the right people around me to challenge, review, and guide the decisions along the way. And to everyone who supported me, reviewed my work, answered questions, or simply trusted me to take this on, thank you. That trust and ownership made this project one of the most meaningful engineering experiences I have had so far.</p><h3>WHAT’S NEXT?</h3><p>This article focused on why I built the platform. The next articles will focus on how I built it.</p><p>I will start from the bare-metal servers and work through the platform layer by layer, including the actual commands, Ansible playbooks, Kubernetes configuration, GPU Operator setup, KAI Scheduler, MIG configuration, JupyterHub, DockerSpawner, resource allocation, persistent storage, shared datasets, and the lifecycle management around user environments.</p><p>In the upcoming articles I wish to share the implementation details, challenges I faced and how I overcame them. Hoping that it will help someone achieve the same or more.</p><h3>FINAL THOUGHTS</h3><p>When I started this project, I was looking at two powerful bare-metal servers with four H200 GPUs. The hardware gave us an enormous amount of compute. But compute by itself does not create a good developer experience.</p><p>The real challenge was turning that compute into something that dozens of people could safely and efficiently use.</p><p>After putting Kubernetes, GPU management, scheduling, MIG, JupyterHub, DockerSpawner, persistent storage, resource isolation, and automated lifecycle management together, those servers became something much more useful. <strong>They became an internal compute platform.</strong></p><p>A developer can open a browser, request resources, get an isolated environment, access the datasets they need, run their experiments, train models, and use GPUs without ever needing to know which physical machine is doing the work.</p><p>And that is ultimately why I built my own Google Colab.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f1140cd5bf9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[25 Git Commands That You’ll Find Here When You Need It The MOST]]></title>
            <link>https://deependraraliya.medium.com/25-git-commands-that-youll-find-here-when-you-need-the-most-de97b73a470d?source=rss-189b213197de------2</link>
            <guid isPermaLink="false">https://medium.com/p/de97b73a470d</guid>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Deependra Raliya]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 21:33:18 GMT</pubDate>
            <atom:updated>2026-08-20T21:34:43.260Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TDgnZQcJhGQTHlRwz89WMg.jpeg" /></figure><p>From college projects, to hobby projects. From being an intern in a startup to being a DevSecOps/ AI infra engineer in a highly regulated industry. I’ve been using Git long enough to have made every possible mistake. I have pushed secrets, nuked histories, merged the wrong branch at 2AM. Over time, you stop googling “how to undo last commit” and start having a personal arsenal. Here are 25 commands and incantations I reach for regularly, grouped by the kind of pain they solve (Total 7 groups).</p><p>For all examples below, imagine we’re working on shopstack, a monorepo ecommerce platform with a Next.js storefront, a Node.js orders API, and a Stripe payments service.</p><h3>Undoing Mistakes</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F5332537&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fguilty-gif-5332537&amp;image=https%3A%2F%2Fmedia.tenor.com%2FSpPUh3E58AoAAAAN%2Fguilty.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/222d3285623dccafd4a051d6771a9462/href">https://medium.com/media/222d3285623dccafd4a051d6771a9462/href</a></iframe><h3>1. Remove last unpushed commit, keep changes staged</h3><pre>git reset --soft HEAD~1<br># or just:<br>git reset HEAD~1</pre><p>--soft is the default when no flag is passed, so git reset HEAD~1 and git reset --soft HEAD~1 are the same thing. The commit disappears, your files stay staged exactly as they were, and you can re-commit whenever you&#39;re ready.</p><p><strong>Example:</strong> You staged all your files and committed. Then immediately noticed the commit message is wrong, or you committed the wrong files by accident:</p><pre>git add .<br>git commit -m &quot;updtae product pge&quot;   # typo, wrong files, whatever</pre><pre>git reset HEAD~1<br># Commit is gone. All your changes are still staged.<br># Now fix whatever was wrong and re-commit.<br>git commit -m &quot;feat: update product listing page&quot;</pre><h3>2. Nuke the last commit entirely</h3><pre>git reset --hard HEAD~1</pre><p>Discards the commit <em>and</em> throws away every file change in it. Nothing survives. ⚠️ This is Destructive. Only use this on local, unpushed commits.</p><p><strong>Example:</strong> You committed something you want to completely throw away. Not stage for later, just gone:</p><pre>git add .<br>git commit -m &quot;wip: experimenting with recommendation sidebar&quot;</pre><pre># Changed your mind, don&#39;t even want the files<br>git reset --hard HEAD~1<br># Working tree is clean. Like the commit never happened.</pre><h3>3. Undo an accidental push without rewriting history</h3><pre>git revert HEAD &amp;&amp; git push</pre><p>Creates a new “undo” commit on top. This is the safe choice for shared branches. Doesn’t rewrite history, just adds an inverse. Use this on main/master instead of a force push. Your teammates will thank you.</p><p><strong>Example:</strong> You pushed a discount code change to main that accidentally made everything 100% off:</p><pre>git revert HEAD<br># Git opens editor: &quot;Revert &#39;feat: apply promo codes at checkout&#39;&quot;<br># Save and close</pre><pre>git push origin main<br># A new revert commit is now on main, site is safe, history intact</pre><h3>4. Force-push after rewriting history (safely)</h3><pre>git push --force-with-lease</pre><p>First, when would you even need --force at all? Any time you <em>rewrite</em> local history, maybe rebasing, amending, resetting, your commit SHAs change. The remote still has the old SHAs. A normal git push will be rejected because git thinks you&#39;re behind. Force-push tells git to ignore that and overwrite the remote with whatever you have locally.</p><p>The problem with bare --force is that it&#39;s blind. If a teammate pushed to the same branch between your last fetch and your force-push, you silently obliterate their commits. Gone. No warning.</p><p>--force-with-lease adds a safety check: it only allows the force-push if the remote tip matches what you last fetched. If someone pushed in the meantime, it refuses, and you fetch first. (You don’t mess up commit histories of other devs)</p><p><strong>Example:</strong> You rebased feature/checkout-v2 on top of latest main and need to update the remote:</p><pre>git rebase main<br># Replayed 4 commits. All SHAs changed.</pre><pre>git push origin feature/checkout-v2<br># rejected remote has old SHAs, git thinks you&#39;re behind</pre><pre>git push --force-with-lease origin feature/checkout-v2<br># Succeeds if nobody else pushed to this branch<br># Fails safely if a teammate pushed while you were rebasing</pre><p>Never use bare --force on any branch someone else might touch. --force-with-lease has zero downside.</p><h3>5. Find the commit that broke something (binary search)</h3><pre>git bisect start<br>git bisect bad<br>git bisect good &lt;sha&gt;</pre><p>Git does a binary search through history. You mark each checkout as good or bad. It narrows to the exact offending commit in O(log n) steps. This is absurdly powerful for regressions in large repos.</p><p><strong>Example:</strong> Order confirmation emails stopped sending sometime in the last 3 weeks. You know v2.1.0 was fine:</p><pre>git bisect start<br>git bisect bad                  # current HEAD is broken<br>git bisect good v2.1.0          # this tag was working</pre><pre># Git checks out the midpoint commit. Test it:<br>node scripts/test-order-email.js</pre><pre>git bisect good   # if it works<br>git bisect bad    # if it&#39;s broken</pre><pre># Repeat ~5 times. Git lands on the exact commit:<br># &quot;b3f9a12 is the first bad commit - refactor: move mailer to queue worker&quot;<br>git bisect reset  # return to HEAD</pre><h3>Recovering Lost Work</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F1720113058264580463&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Ffaceless-faceless-tv-masked-creator-masked-guy-neon-green-ski-mask-gif-1720113058264580463&amp;image=https%3A%2F%2Fmedia.tenor.com%2FF98RtoNfAW8AAAAN%2Ffaceless-faceless-tv.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd409dbf0e40533e074d0d4d70dba0ed/href">https://medium.com/media/bd409dbf0e40533e074d0d4d70dba0ed/href</a></iframe><h3>6. Resurrect a dropped stash or deleted branch</h3><pre>git reflog</pre><p>Every HEAD movement is logged here. Did a hard reset nuke your work? Find the SHA in reflog and git checkout &lt;sha&gt;. This has saved me more than once at 11pm. Learn this before you need it.</p><p><strong>Example:</strong> You accidentally deleted the feature/wishlist branch before merging it:</p><pre>git branch -D feature/wishlist   # oops</pre><pre>git reflog<br># HEAD@{0}: checkout: moving from feature/wishlist to main<br># HEAD@{1}: commit: feat: add wishlist persistence to Redis<br># HEAD@{2}: commit: feat: wishlist UI components</pre><pre>git checkout -b feature/wishlist HEAD@{1}<br># Branch restored, all commits intact</pre><h3>7. Recover a specific file from a past commit</h3><pre>git checkout &lt;sha&gt; -- path/to/file</pre><p>Checks out just that file from that commit into your working tree. No branch switch, no drama.</p><p><strong>Example:</strong> You accidentally deleted or overwrote the Stripe webhook handler:</p><pre>git log --oneline -- services/payments/webhook.ts<br># a1b2c3d feat: stripe webhook handler</pre><pre>git checkout a1b2c3d -- services/payments/webhook.ts<br># File is back in your working tree, ready to stage and commit</pre><h3>8. Grab a single commit from another branch</h3><pre>git cherry-pick &lt;sha&gt;</pre><p>Applies one specific commit onto your current branch, without merging the whole branch.</p><p><strong>Example:</strong> A critical oversell fix was committed to feature/inventory-v2, which isn&#39;t ready to ship. You need just that fix on main today:</p><pre>git log feature/inventory-v2 --oneline<br># e7f8a9b fix: prevent oversell on concurrent checkout   ← this one<br># d6c5b4a feat: new inventory sync UI (not ready)</pre><pre>git switch main<br>git cherry-pick e7f8a9b<br>git push origin main<br># Just the fix, none of the unfinished work</pre><h3>Cleaning Up History</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F22258443&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fspiderman2-peter-parker-broom-closet-janitor-closet-mops-gif-22258443&amp;image=https%3A%2F%2Fmedia.tenor.com%2FFwGk51cxUxUAAAAN%2Fspiderman2-peter-parker.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/6735163cfeeae9040b0a944af5a54b65/href">https://medium.com/media/6735163cfeeae9040b0a944af5a54b65/href</a></iframe><h3>9. Interactive rebase (squash, reorder, rewrite)</h3><pre>git rebase -i HEAD~N</pre><p>Here’s what interactive rebase actually does that makes it worth understanding deeply: it lets you go back in time and rewrite your last N commits before they become part of shared history. Think of it as “edit mode” for your commit log.</p><p>When you open it, git lists your last N commits in a text editor. Each line starts with a command word. pick by default, meaning &quot;keep this commit as-is&quot;. You change those command words to change what happens:</p><ul><li>squash: fold this commit into the one above it (combine them into one)</li><li>reword: keep the changes, but let me edit the message</li><li>drop: delete this commit entirely</li><li>edit: pause here so I can amend this commit before continuing</li></ul><p>The practical use case: your feature branch looks like wip, wip, fix typo, actually fix it, more fixes, done. Before you open a PR, you squash all of that into one clean, meaningful commit.</p><p><strong>Example:</strong> Your product filter branch has 5 messy commits. You want to merge it as 2 clean ones:</p><pre>git rebase -i HEAD~5</pre><pre># Editor opens - change the command words:<br># pick 1a2b3c feat: product filter UI<br># squash 4d5e6f wip: filter logic<br># squash 7g8h9i fix typo in filter<br># squash 0j1k2l actually working now<br># pick 6p7q8r feat: persist filters in URL params</pre><pre># Git combines commits 1-4 into one, leaves commit 5 separate.<br># Opens editor again to let you write the combined commit message.<br># Result: 2 clean commits instead of 5 messy ones.</pre><p>One important thing: rebase rewrites SHAs. If these commits are already pushed, you’ll need --force-with-lease afterward.</p><h3>10. Amend the last commit (message or content)</h3><pre>git commit --amend --no-edit</pre><p>What --amend actually does: it takes your currently staged changes, adds them to the previous commit, and creates a <em>brand new commit object</em> to replace it. The old commit is discarded. The new one has a new SHA. This is why you shouldn&#39;t amend commits that are already pushed to a shared branch. You&#39;d rewrite a commit that other people may have based their work on.</p><p>--no-edit means &quot;keep the existing commit message&quot;. Without it, git opens the editor for you to change the message too.</p><p><strong>Example:</strong> You committed, then immediately realized you forgot to stage one file:</p><pre>git add .<br>git commit -m &quot;feat: add product image zoom&quot;</pre><pre># Then notice you forgot to stage zoom.css<br>git add src/styles/zoom.css<br>git commit --amend --no-edit</pre><pre># What git did internally:<br># - Took the previous commit (SHA: abc123)<br># - Added zoom.css to it<br># - Created a completely new commit (SHA: def456) with both changes<br># - abc123 is now orphaned and will eventually be garbage collected<br># The history shows one clean commit, as if you staged everything correctly the first time</pre><p>Fixing just the message (no staged changes needed):</p><pre>git commit --amend -m &quot;feat: product image zoom with pinch support&quot;</pre><h3>11. Permanently remove a file from all of history</h3><p>Have you ever pushed an API key that is active in production into your commit? You may think that nobody will find it, or I’ll make another commit to hide it. But it’ll be buried somewhere in the commit histories to be grepped by hackers. It’s very to scrape a repo for leaked secrets and abuse your APIs even for non-technical people especially in 2026.</p><p>Here’s how you can delete that commit or a specific file from it entirely. No trace that it ever existed.</p><pre># You&#39;ll have to install it first. It doesn&#39;t come WITH git:<br>pip install git-filter-repo</pre><pre>git filter-repo --path .env.production --invert-paths</pre><p>git filter-repo is a separate Python tool, not a built-in git command. It replaces the old git filter-branch which was slow, dangerous, and officially discouraged. --invert-paths means &quot;rewrite history <em>without</em> this file&quot;. It walks every single commit and removes the file from each one. ⚠️ This is a nuclear level threat if you don’t know what you’re doing.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F20846678&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Frowoon-panic-panicking-nervous-sketchy-gif-20846678&amp;image=https%3A%2F%2Fmedia.tenor.com%2FCi9T7kzTF7QAAAAN%2Frowoon-panic.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/daf1271bc05c2780a59b66c84cac91b4/href">https://medium.com/media/daf1271bc05c2780a59b66c84cac91b4/href</a></iframe><p>Coordinate with your team first, everyone needs to re-clone after.</p><p><strong>Example:</strong> A dev committed .env.production with the live Stripe secret key 3 commits ago:</p><pre>pip install git-filter-repo</pre><pre>git filter-repo --path .env.production --invert-paths<br># Rewrites all commits - file is gone from entire history</pre><pre>git push origin --force --all<br>git push origin --force --tags</pre><pre># NOW: immediately rotate the Stripe key and DB password<br># Notify team: everyone must re-clone, local copies still have the old history</pre><h3>12. Squash an entire branch into one commit</h3><pre>git merge --squash feature-branch &amp;&amp; git commit</pre><p>Takes all the changes from a branch and stages them as one new commit. Clean merge history without 30 WIP commits in your log.</p><p><strong>Example:</strong> feature/returns-portal has 23 commits over 2 weeks. You want one clean entry in main:</p><pre>git switch main<br>git merge --squash feature/returns-portal<br>git commit -m &quot;feat: customer returns and refund portal (#412)&quot;<br># main gets one atomic commit</pre><h3>Branching &amp; Merging</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F12662834225987595987&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Ffarm-merge-merge-farm-merge-valley-merging-discord-merge-gif-12662834225987595987&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fr7tso1GmLtMAAAAN%2Ffarm-merge-merge.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/e79e87575931deb4135b7139cb226cfe/href">https://medium.com/media/e79e87575931deb4135b7139cb226cfe/href</a></iframe><h3>13. Create and switch to a new branch in one shot</h3><pre>git switch -c feature/my-thing</pre><p>git checkout was the original all-purpose command for both switching branches <em>and</em> restoring files. This caused real confusion. git checkout main switches your branch, but git checkout -- file.ts discards file changes. Same command, completely different behaviors depending on the argument.</p><p>Git 2.23 split this into two purpose-built commands: git switch for branch operations, git restore for file operations. switch is now the correct way to change branches. That’s why the intent is unambiguous, and it won&#39;t accidentally discard file changes if you mistype.</p><pre>git switch -c feature/abandoned-cart-emails   # create + switch<br>git switch main                               # switch to existing<br>git switch -                                  # back to previous branch</pre><p>checkout still works, but switch is clearer for anything branch-related.</p><h3>14. Rebase instead of merge to keep linear history</h3><pre>git pull --rebase origin main</pre><p>Replays your local commits on top of the updated remote instead of creating a merge commit. Set pull.rebase=true globally and forget about it.</p><p><strong>Example:</strong> You’ve been working on the order tracking page while 4 other PRs merged into main:</p><pre>git pull --rebase origin main<br># Rewinding head to replay your work on top...<br># Applying: feat: order tracking timeline component<br># Applying: feat: carrier API integration<br># Linear history - no &quot;Merge branch &#39;main&#39; into feature/order-tracking&quot; noise</pre><h3>15. Delete a remote branch</h3><pre>git push origin --delete branch-name</pre><p>Cleans up after a merged PR.</p><p><strong>Example:</strong></p><pre>git push origin --delete feature/checkout-v2</pre><pre># Sync the deletion locally and for teammates:<br>git fetch --prune<br># x [deleted] origin/feature/checkout-v2</pre><h3>16. Stash only staged changes (leave unstaged alone)</h3><pre>git stash push --staged</pre><p>Surgical stashing. Stashes only what’s staged, leaves your unstaged WIP files untouched.</p><p><strong>Example:</strong> You’re mid-refactor with some clean staged changes and some messy unstaged edits. A P1 drops:</p><pre># Staged: clean changes to orders/index.ts<br># Unstaged: half-finished edits to orders/filters.ts</pre><pre>git stash push --staged<br># Only staged changes go into the stash. Unstaged files stay.</pre><pre>git switch hotfix/payment-timeout<br># Fix, push, deploy</pre><pre>git switch feature/orders-refactor<br>git stash pop<br># Staged changes are back. Unstaged files were never touched.</pre><h3>Inspection &amp; Search</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F7355945525275031653&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fit-wasn%27t-me-wasn%27t-me-benjammins-defensive-hands-up-gif-7355945525275031653&amp;image=https%3A%2F%2Fmedia.tenor.com%2FZhWRe1UkeGUAAAAN%2Fit-wasn%2527t-me-wasn%2527t-me.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e073a133678bb1cd9a16c1f868e09ec/href">https://medium.com/media/1e073a133678bb1cd9a16c1f868e09ec/href</a></iframe><h3>17. Find which commit introduced a string</h3><pre>git log -S &quot;search_string&quot; --source --all</pre><p>The -S flag is called the &quot;pickaxe&quot;. Unlike git grep which searches the <em>current</em> file contents, -S searches through the <em>diff history. I</em>t finds commits where the number of occurrences of that string changed. Meaning it finds the exact commit where that string was added or removed from the codebase.</p><p>This is the command for questions like: “when did this function appear?”, “which commit deleted this env variable?”, “who introduced this API key?”.</p><p><strong>Example:</strong> Free shipping stopped working. You suspect the threshold constant was renamed or deleted. Find exactly when it was touched:</p><pre>git log -S &quot;SHIPPING_FREE_THRESHOLD&quot; --source --all --oneline<br># 9f3a21c (origin/feature/shipping-v2) refactor: centralise shipping config<br># 4b1d88e feat: add free shipping threshold env var</pre><pre># Two results: first it was added (4b1d88e), then it was changed (9f3a21c)<br># Inspect what the refactor actually did to it:<br>git show 9f3a21c -- services/shipping/config.ts</pre><p>You can also use -G instead of -S if you want to search by regex rather than exact string.</p><h3>18. Show a pretty, graph-style log</h3><pre>git log --oneline --graph --all --decorate</pre><p>The visual branch graph in your terminal. I alias this to git lg. Understand the actual state of a repo in one glance before doing anything destructive.</p><p><strong>Example:</strong></p><pre>git log --oneline --graph --all --decorate</pre><pre># * 3a1b2c3 (HEAD -&gt; main, origin/main) feat: order tracking timeline<br># | * f9e8d7c (origin/feature/returns-portal) feat: refund status page<br># | * a1b2c3d feat: returns form validation<br># |/<br># * 9b8a7c6 (tag: v2.3.0) chore: release v2.3.0</pre><p>Add to ~/.gitconfig:</p><pre>[alias]<br>  lg = log --oneline --graph --all --decorate</pre><h3>19. Blame with whitespace ignored and moved code tracked</h3><pre>git blame -w -C -C -C &lt;file&gt;</pre><p>Plain git blame shows you who last touched each line. But it has two major failure modes in real codebases:</p><p><strong>Failure mode 1: whitespace changes.</strong> Someone runs prettier or reformats the file. Now git blame shows them as the author of every single line they reformatted, even if they never changed the logic. -w ignores whitespace-only changes so formatting commits don&#39;t steal authorship.</p><p><strong>Failure mode 2: code that moved.</strong> In any mature codebase, code gets reorganized. A function written in services/tax/calculator.ts gets moved to src/checkout/summary.ts during a refactor. Plain git blame on summary.ts will show the person who did the refactor as the author of those lines. They just moved it, they didn&#39;t write it.</p><p>This is where -C comes in. A single -C detects lines moved within the same commit. Two -C -C also detects lines moved from files modified in the same commit. Three -C -C -C goes furthest. It searches the entire history and traces lines back to their original source file, even if that file no longer exists.</p><p><strong>Example:</strong> The tax calculation logic in checkout/summary.ts is wrong. Who actually wrote it?</p><pre># Plain blame - unhelpful<br>git blame src/checkout/summary.ts<br># priya   src/checkout/summary.ts  (2024-11-03)  const tax = amount * rate;<br># priya   src/checkout/summary.ts  (2024-11-03)  return total + tax;<br># Blames priya - she just moved it in a refactor</pre><pre># With full flags<br>git blame -w -C -C -C src/checkout/summary.ts<br># james   services/tax/calculator.ts  (2024-08-14)  const tax = amount * rate;<br># james   services/tax/calculator.ts  (2024-08-14)  return total + tax;<br># James wrote it in the original file 3 months ago. Talk to James.</pre><p>The tradeoff: -C -C -C is noticeably slower on large repos because it has to scan all history. On a file you&#39;re actively debugging, it&#39;s worth it.</p><h3>20. Show exactly what a PR will change</h3><pre>git diff main...feature --stat</pre><p>Three dots shows the diff from the common ancestor (exactly what the PR adds). Two dots would include main’s recent commits too. Know the difference.</p><p><strong>Example:</strong></p><pre>git diff main...feature/multi-currency --stat<br># src/checkout/pricing.ts         | 84 ++++++++++---<br># services/fx-rates/client.ts     | 156 +++++++++++++++++++++<br># 5 files changed, 357 insertions(+), 23 deletions(-)</pre><pre>git diff main...feature/multi-currency -- src/checkout/pricing.ts<br># Drill into a specific file</pre><h3>Releases &amp; Changelogs</h3><h3>21. Generate a changelog between two tags</h3><pre>git log v1.2.0..v1.3.0 --pretty=format:&quot;- %s (%an)&quot; --no-merges</pre><p><strong>Example:</strong></p><pre>git log v2.3.0..v2.4.0 --pretty=format:&quot;- %s (%an)&quot; --no-merges<br># - feat: customer returns and refund portal (sarah)<br># - feat: order tracking timeline component (james)<br># - fix: prevent oversell on concurrent checkout (james)</pre><pre># Filter to only features if using Conventional Commits:<br>git log v2.3.0..v2.4.0 --pretty=format:&quot;- %s&quot; --no-merges --grep=&quot;^feat&quot;</pre><h3>22. Tag a release with a message</h3><pre>git tag -a v2.4.0 -m &quot;Release v2.4.0&quot; &amp;&amp; git push --tags</pre><p>Annotated tags carry metadata like tagger, date, message. Required if your CI/CD pipeline triggers on tags.</p><p><strong>Example:</strong></p><pre>git switch main &amp;&amp; git pull<br>git tag -a v2.4.0 -m &quot;Release v2.4.0 returns portal, order tracking&quot;<br>git push --tags<br># GitHub Actions triggers on tag push, builds shopstack:v2.4.0, deploys to prod</pre><h3>23. Count commits per author</h3><pre>git shortlog -sn --no-merges</pre><p><strong>Example:</strong></p><pre>git shortlog -sn --no-merges v2.3.0..v2.4.0<br>#    34  sarah<br>#    28  james<br>#    19  priya<br>#     6  devops-bot</pre><h3>Maintenance &amp; Housekeeping</h3><h3>24. Remove all untracked files and directories</h3><pre>git clean -fdx</pre><p>Wipes untracked files (-f), directories (-d), and gitignored files (-x). ⚠️ Always dry-run with -n first.</p><p><strong>Example:</strong> Build passing locally, failing on CI. Suspect stale artifacts:</p><pre>git clean -ndx<br># Would remove node_modules/, .next/, dist/</pre><pre>git clean -fdx<br>npm install &amp;&amp; npm run build<br># Clean slate. Confirms whether it&#39;s an environment issue</pre><h3>25. Prune stale remote refs and compress the object store</h3><pre>git fetch --prune &amp;&amp; git gc --aggressive</pre><p><strong>Example:</strong> shopstack is 2 years old, hundreds of dead branches, and git status feels slow:</p><pre>git fetch --prune<br># x [deleted] origin/feature/checkout-v1<br># x [deleted] origin/hotfix/stripe-webhook-2023<br># ... 40 more stale refs pruned</pre><pre>git gc --aggressive<br># Repo: 340MB → 180MB. Commands feel instant again.</pre><p>Most of these I learned right after I needed them (which is the worst time to learn anything)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F27690748&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fgreenhairbil-sus-dog-gif-27690748&amp;image=https%3A%2F%2Fmedia.tenor.com%2FahioZMimaREAAAAN%2Fgreenhairbil-sus-dog.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/56d549098f14f17ae6a93f8279cb0865/href">https://medium.com/media/56d549098f14f17ae6a93f8279cb0865/href</a></iframe><p>The three worth committing to muscle memory before anything else: reflog (your universal safety net), filter-repo (your credential incident response), and bisect (your debugging superpower). The rest will come naturally once you&#39;ve been burned enough times.</p><p>Happy committing.</p><p>Do let me know which ones helped you out and when did you use it?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de97b73a470d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Actually Moved 20TB of Live AI Production Data: The Complete Technical Breakdown]]></title>
            <link>https://deependraraliya.medium.com/how-i-actually-moved-20tb-of-live-ai-production-data-the-complete-technical-breakdown-b393823f3e46?source=rss-189b213197de------2</link>
            <guid isPermaLink="false">https://medium.com/p/b393823f3e46</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Deependra Raliya]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 19:45:01 GMT</pubDate>
            <atom:updated>2026-06-15T19:48:38.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*b5HD0T2EEzPx-Xvr.png" /></figure><p>The source was a live Kubernetes node — 12 projects, 100+ containers, 8 H100 GPUs running model training and inference, a Qdrant vector store, traditional databases, NFS mounts, CI/CD agents, and everything accumulated over the server’s lifetime. 20TB+. All of it had to arrive at the destination exactly as it left the source — ownership, permissions, ACLs, extended attributes, hard links, and all — while the production environment kept running undisturbed on the same machine.</p><p>This is the exact command I built, flag by flag, and why each one is there.</p><h3>What Was on That Server</h3><p>Before the tooling, it helps to understand the surface area. This wasn’t a file store. It was a full production Kubernetes node with:</p><ul><li>Application-generated output, processed results, and job artifacts</li><li>Raw and curated datasets used by ML and data pipelines</li><li>Container image layers and Docker data</li><li>Kubernetes manifests and ConfigMaps</li><li>Docker volumes and bind-mount directories tied to running containers</li><li>nginx configurations, TLS certificates, and virtual host definitions</li><li>Service logs and structured log archives</li><li>Cron jobs and automation scripts</li><li>Qdrant vector store and traditional database data</li><li>Shared NFS-mounted directories used across deployments</li><li>CI/CD agent workspaces and caches</li><li>Monitoring agent configs and collected metrics data</li></ul><p>Some of those paths had ownership structures that running services depended on. Some had ACLs and extended attributes set by security tooling. Everything had to land at the destination in exactly the same state.</p><h3>The Non-Negotiable Constraints</h3><p>Before selecting any tool or writing any command, I defined what the migration absolutely had to satisfy:</p><ol><li><strong>Run in the background: </strong>the source was live, it couldn’t be taken offline</li><li><strong>Be resumable: </strong>a multi-day transfer over SSH will be interrupted; it needed to pick up exactly where it left off</li><li><strong>Never corrupt data on interruption: </strong>a half-written file that looked complete was worse than a missing file</li><li><strong>Not compete with production for resources: </strong>disk I/O, bandwidth, and RAM were shared with live services</li><li><strong>Preserve all metadata exactly: </strong>ownership, permissions, ACLs, xattrs, hard links, timestamps</li><li><strong>Survive a terminal disconnect: </strong>the process couldn’t be tied to a shell session</li></ol><p>Every flag in the final command is a direct answer to one of these constraints.</p><h3>Building the Command, One Problem at a Time</h3><h3>Starting at the Naive Baseline</h3><p>The simplest possible thing:</p><pre>rsync /source_folder/ user@dest_ip:/dest_folder/</pre><p>This copies file content. That’s it. Everything else like metadata, ownership, resumability, resource throttling, etc is missing. Here’s how I added each piece.</p><h3>Problem 1: Permissions and Ownership Will Be Lost</h3><p>A plain rsync copies bytes but drops metadata. On a Kubernetes host, containers expect specific UIDs on their bind-mount directories. nginx expects to own its config files. An application configured to run as a specific service account will fail to read its own data if ownership is wrong.</p><p><strong>What could go wrong:</strong> Containers fail to mount volumes. Services refuse to read config files. Setuid binaries silently stop working. You spend hours on forensic permission debugging on a machine that was supposed to be a clean copy.</p><p>The fix is -a - archive mode. One flag that enables recursive copy, symlink preservation, permission preservation, ownership preservation, and timestamp preservation together.</p><pre>rsync -a /source_folder/ user@dest_ip:/dest_folder/</pre><h3>Problem 2: ACLs and Extended Attributes Will Be Silently Dropped</h3><p>Archive mode handles standard Unix permissions, but not ACLs or extended attributes (xattrs). Security tooling, SELinux labels, Linux capabilities on binaries, and app-specific metadata live in xattrs. -a doesn&#39;t touch any of this.</p><p><strong>What could go wrong:</strong> A binary with CAP_NET_BIND_SERVICE set via xattrs loses that capability on the destination. A file with an ACL granting a service account read access falls back to standard permissions and gets denied. Both fail silently until something tries to use them.</p><p>-A preserves ACLs. -X preserves extended attributes. Both are required independently - neither is implied by -a.</p><pre>rsync -a -A -X /source_folder/ user@dest_ip:/dest_folder/</pre><h3>Problem 3: Hard Links Will Become Duplicate Files</h3><p>Hard links are multiple directory entries pointing to the same inode — the same bytes on disk, no duplication. Without -H, rsync treats each hard link as a separate file and copies the content independently. On a server with Docker image layers and package managers that use hard links heavily, this silently inflates disk usage on the destination.</p><p><strong>What could go wrong:</strong> The destination disk fills unexpectedly mid-transfer because what was 20TB of hard-linked data is now meaningfully larger. Deduplication and backup tooling downstream also behaves incorrectly because the inode structure is different.</p><p>-H detects hard links on the source and recreates them as hard links on the destination.</p><pre>rsync -a -A -X -H /source_folder/ user@dest_ip:/dest_folder/</pre><h3>Problem 4: Service Account UIDs May Not Match on the Destination</h3><p>By default, rsync resolves UIDs and GIDs to usernames before transferring, then re-resolves them on the destination using /etc/passwd. For AD/Kerberos domain users, this is consistent - UIDs are derived from the AD domain via SSSD or winbind, so the same user maps to the same UID on both servers.</p><p>But local service accounts — nginx, www-data, postgres, docker, prometheus - are assigned UIDs at package install time. On a freshly provisioned destination where packages were installed in a different order, nginx may have UID 995 instead of 998. rsync&#39;s name resolution will correctly map files owned by nginx to nginx on the destination - but only if nginx exists there.</p><p><strong>The right approach:</strong> Pre-create all local service accounts on the destination with the same UIDs as the source <em>before</em> running the transfer. Let rsync resolve by username (the default behaviour). If a username doesn’t exist on the destination, rsync falls back to the numeric UID — which becomes a visible warning that something needs to be created, not a silent mismatch.</p><p>This is more reliable than --numeric-ids in an AD environment. Domain user UIDs are already consistent. Service account UIDs need to be explicitly matched by pre-creating the accounts - not bypassed by transferring raw numbers and hoping they land on the right user.</p><pre>rsync -a -A -X -H /source_folder/ user@dest_ip:/dest_folder/<br># (no change to the command - the fix is operational, not a flag)</pre><h3>Problem 5: A Network Drop or SSH Timeout Kills the Transfer</h3><p>20TB over SSH is not a five-minute job. Network hiccups happen. SSH sessions time out. If rsync exits mid-transfer without partial-file protection, the destination has half-written files that look complete. The next run may skip them (file exists), and you discover corruption only when an application tries to read the data.</p><p><strong>What could go wrong:</strong> A partially transferred 50GB dataset file sits on the destination appearing intact. The transfer “completes” on the next run. Corruption discovered in production, not in the transfer.</p><p>--partial keeps interrupted files on disk so rsync can resume them. --partial-dir=.rsync-partial is more precise - it stages incomplete files in a named directory so they never appear in-place as fake-complete files. On resume, rsync finds and continues from the staging directory.</p><pre>rsync -a -A -X -H \<br>  --partial --partial-dir=.rsync-partial \<br>  /source_folder/ user@dest_ip:/dest_folder/</pre><h3>Problem 6: The Transfer Will Compete With Production for Disk I/O</h3><p>rsync reading continuously from a production disk puts sustained pressure on the I/O queue. Without throttling, it competes with live services for the same queue — database writes, container disk reads, log flushes. Latency spikes. Timeouts follow.</p><p><strong>What could go wrong:</strong> A Kubernetes pod’s database write takes 10x longer because rsync is saturating the I/O queue. Services start throwing timeouts. Alerts fire. The migration that was supposed to be invisible to production is now an incident.</p><p>ionice wraps rsync and assigns it an I/O scheduling class before it touches any file. Class 3 (idle) means the process only gets disk access when no other process wants it - the most conservative setting. For slightly more throughput with still-low impact, class 2 (best-effort) at niceness 7 is a reasonable middle ground.</p><p>This wraps around the whole command — the final shape is taking form now.</p><h3>The Final Pipeline</h3><p>Two distinct commands, each tuned for a different category of data. Both ran inside tmux (covered below).</p><h3>Command 1: Compressible Data (configs, logs, text)</h3><pre>ionice -c3 sudo rsync -avzP \<br>  --partial-dir=.rsync-partial \<br>  -e &#39;ssh -l &quot;user@aragen.com&quot;&#39; \<br>  /source_folder/ \<br>  dest_ip:/dest_folder</pre><p>Flag What it does ionice -c3 Idle I/O class - disk access only when no other process wants it. The safest setting for a live production host. sudo Required to read files owned by system users and to preserve original ownership on transfer. -a Archive mode: recursive copy + symlink, permission, ownership, and timestamp preservation in one flag. -v Verbose - shows each file being transferred in real time. -z Compresses data in transit. Significant bandwidth reduction for text-heavy content like logs and configs. -P Combines --partial (keep interrupted files) and --progress (per-file progress). The core of resumability. --partial-dir Stages incomplete files in a named directory. They never appear in-place as fake-complete files. -e &#39;ssh ...&#39; Specifies the remote shell and login user. Key-based SSH authentication.</p><h3>Command 2: Binary Data (datasets, Docker volumes, binaries)</h3><pre>sudo ionice -c2 -n7 rsync -aAXHv \<br>  --partial \<br>  --info=progress2 \<br>  -e &#39;ssh -l &quot;user@aragen.com&quot;&#39; \<br>  /source_folder/ dest_ip:/dest_folder/</pre><p>Flag What it does ionice -c2 -n7 Best-effort I/O class at lowest priority (niceness 7). More throughput than idle class, still heavily deprioritized vs normal processes. -A Preserves ACLs - extended permission entries beyond standard user/group/other. -X Preserves extended attributes - security labels, capabilities, app-specific metadata. -H Preserves hard links. Without this, hard-linked files become independent copies, inflating disk usage. --partial Keeps partially transferred files on interruption for resuming. --info=progress2 Overall transfer stats (total size, bytes transferred, ETA) instead of per-file progress. Cleaner for jobs with millions of small files.</p><p><strong>Why two commands instead of one:</strong> Compression (-z) is counterproductive for already-compressed or binary data - it burns CPU cycles with no bandwidth benefit. Command 1 optimises for bandwidth efficiency on text-heavy content. Command 2 optimises for full metadata fidelity on datasets and binaries where every byte and every ACL matters.</p><h3>Problem 7: A Terminal Disconnect Will Kill the Session</h3><p>SSH sessions are tied to a terminal process. A VPN drop, a closed laptop, a network blip — the shell exits, rsync receives SIGHUP, and the transfer dies. Even with --partial, you still lose whatever was in-flight and have to restart the run manually.</p><p>Both commands ran inside a tmux session on the source server. The process is owned by tmux, not the SSH terminal - disconnecting from SSH leaves it running. Reconnect, reattach, and the transfer output is exactly where you left it.</p><pre>tmux new -s migration<br># run the ionice + rsync command inside<br># detach any time with Ctrl+B, D</pre><pre># reconnect from anywhere:<br>tmux attach -t migration</pre><h3>The Dry Run and Test Passes</h3><p>Before touching any production directory at scale, I ran --dry-run to verify the file list, confirm ownership resolution looked correct, and validate that SSH key auth against the destination was working cleanly. Then I ran the real command on a small non-critical directory and watched it in htop and iotop - confirming that ionice was actually deprioritizing the process in the I/O queue and that the resource profile on the source was flat.</p><blockquote><em>A dry run costs nothing. Discovering a misconfigured ownership flag after 48 hours of transfer costs everything.</em></blockquote><p>Only after those sanity checks passed did I run the full pipeline.</p><h3>What This Toolchain Solves, End to End</h3><p>Constraint Solution Run in background, invisible to production ionice -c3 / -c2 -n7 Survive terminal disconnect tmux Resume after interruption --partial, --partial-dir Never corrupt data on interruption --partial-dir (staged incomplete files) Preserve permissions and ownership -a Preserve ACLs and extended attributes -A, -X Preserve hard links -H Correct ownership on destination Pre-create service accounts with matching UIDs Encrypted transport SSH Bandwidth efficiency for text data -z</p><p>Four tools. Seven deliberate flags. One uneventful migration.</p><p><em>For the story behind this — how I got the responsibility, what I was thinking before touching any command, and what I learned about thinking like a platform engineer — read </em><a href="https://claude.ai/chat/migration_part1.md"><em>Part 1</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b393823f3e46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating a 20TB Live Production Server: Planning, Ownership, and the Engineering Mindset Behind It]]></title>
            <link>https://deependraraliya.medium.com/migrating-a-20tb-live-production-server-planning-ownership-and-the-engineering-mindset-behind-it-90e55e086556?source=rss-189b213197de------2</link>
            <guid isPermaLink="false">https://medium.com/p/90e55e086556</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[server-migration]]></category>
            <dc:creator><![CDATA[Deependra Raliya]]></dc:creator>
            <pubDate>Sat, 13 Jun 2026 10:43:45 GMT</pubDate>
            <atom:updated>2026-06-13T10:43:45.147Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZTBd6kbVbtbc8H0Rz-xOBA.png" /></figure><p>Six months into my first role as a DevOps engineer, I owned the end-to-end migration of a live production bare-metal server.</p><p>20TB of data, 100+ containers across 12 projects, 8 H100 GPUs running model training and inference pipelines that had to be migrated from Yotta’s cloud infrastructure to our own on-prem data center. The source server stayed live throughout. Optimal maintenance window. No downtime.</p><p>This post is about what happened before the first command ran.</p><h3>What the Server Actually Was</h3><p>This was not a file server or a staging environment. It was a production Kubernetes node — the kind where things break in ways you don’t fully anticipate until they do.</p><p>Running on it at any given moment:</p><ul><li>100+ containers running live traffic and Data/AI workloads across more than 12 projects</li><li>8 H100 GPUs running model training and inference pipelines</li><li>A Qdrant vector store and specialized production data stores</li><li>Docker volumes and bind-mount directories with ownership structures that services actively depended on</li><li>nginx configurations, TLS certificates, and virtual host definitions</li><li>NFS-mounted shared directories used across multiple deployments</li><li>CI/CD agents, cron jobs, and automation scripts</li><li>Monitoring agents, structured log archives, and metrics data</li><li>ML datasets, job artifacts, and over 20TB of accumulated application data</li></ul><p>The destination was a brand new on-prem machine — completely cold, no services running. Everything on the source had to arrive at the destination exactly as it left: ownership, permissions, ACLs, extended attributes, hard links, timestamps. No exceptions.</p><p>The fallback, if things went badly enough, was a week-old full server backup — meaning 6–7 days of production data lost, plus the time spent on the migration itself, plus cascading issues I couldn’t fully model in advance. That was the real cost of getting this wrong.</p><h3>Before Touching Anything: Building the Migration Plan</h3><p>The first thing I did was treat this as an engineering problem, not an operational task. The distinction matters. An operational task has a known procedure — you follow it. An engineering problem requires you to first understand the system well enough to design the procedure yourself.</p><p>I had spent the previous six months documenting the environment in detail — every application, its dependencies, its DNS records, its database connections, the ownership structure of its data directories. That documentation wasn’t created for this migration, but it became the foundation of it.</p><p>From that base, I built the migration plan before proposing to own the work. It covered:</p><p><strong>Inventory and dependency mapping:</strong> Every application on the source, mapped to its dependencies such as databases, volumes, other services. This determined migration order as you can’t bring up a service before its dependencies exist on the destination, and in a 12-project environment with shared infrastructure, that ordering is non-trivial.</p><p><strong>Ownership and permission audit:</strong> For every directory with non-standard ownership or ACLs, a record of what it should look like on the destination. Service accounts like nginx, postgres, application-specific users needed to exist on the destination before the data transfer ran. Otherwise ownership resolution would be problematic if all files were owned by a single user account who ran the transfer.</p><p><strong>Failure scenario documentation:</strong> For each phase of the migration from pre-migration, “during transfer” and post-cutover, a documented response for the most likely failure modes. What does a mid-transfer interruption look like, and how is it recovered? What does a service that fails to start on the destination indicate, and what’s the diagnostic path? What’s the rollback trigger, the specific condition under which we accept data loss and restore from backup rather than continuing?</p><p><strong>Validation checklist: </strong>A post-migration checklist for each application like connectivity, data integrity checks, log output. Migrating data is not the same as migrating a working system. The checklist is what closes that gap.</p><p>This plan existed before I was assigned the task. When I presented it, the response was straightforward. <strong>Take it on yourself. </strong>Preparation had created the opportunity , there was no compelling reason to hand it to someone else.</p><h3>The Single Constraint That Shaped Everything</h3><p>Once I owned the work, the first technical question wasn’t “which tool do I use.” It was, <em>what are the non-negotiable constraints, and what do they rule out?</em></p><p>The source server was live throughout the migration. That one fact cascaded into everything:</p><ul><li>The migration had to run as a true background process, no resource spikes, no I/O saturation, no bandwidth consumption that production services would feel</li><li>It had to be interruptible and resumable, a multi-day transfer would be interrupted, and restarting from scratch was not acceptable</li><li>Interrupted transfers could not leave corrupt data on the destination, a half-written file that looked complete will be a worse outcome than a missing file</li><li>The process had to be capable of surviving terminal disconnects, it couldn’t be tied to a shell session that a VPN drop or a closed laptop could kill</li></ul><p>Every tool choice and every flag in the final command is a direct answer to one of these constraints. That’s the sequence that matters: constraints first, tooling second. The flags aren’t arbitrary, they’re the implementation of decisions already made on paper.</p><h3>What This Kind of Work Actually Requires</h3><p>Looking back, the reason this migration went cleanly wasn’t the toolchain. The toolchain was the easy part: rsync, ionice, tmux, SSH. All documented, all standard.</p><p><strong>What made it work was the time spent before touching any of it: </strong>Understanding the system deeply enough to know what could go wrong, documenting it specifically enough that every failure mode had a defined response, and not starting the transfer until the destination environment was in a state where the transfer could actually succeed.</p><p>That’s the pattern I kept returning to throughout this project, and the one I’d apply to any infrastructure work of this kind. The quality of the execution is almost entirely determined by the quality of the preparation. A migration plan with dependency maps, ownership audits, failure scenarios, and validation checklists is not overhead. It’s the work.</p><p>This post was about my planning and preparation of my first ever large scale production event.</p><p><em>Part 2 of this covers the technical execution in full, the toolchain, the failure modes that shaped each flag, and the rsync command built from scratch, one problem at a time.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90e55e086556" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>