<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amantha Sandupa on Medium]]></title>
        <description><![CDATA[Stories by Amantha Sandupa on Medium]]></description>
        <link>https://medium.com/@amanthaa?source=rss-eb26b525a52e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Q6nbZz6xLm2AZ9E5</url>
            <title>Stories by Amantha Sandupa on Medium</title>
            <link>https://medium.com/@amanthaa?source=rss-eb26b525a52e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:52:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amanthaa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Your Git Repo Needs Git LFS for Large Files like ML models]]></title>
            <link>https://amanthaa.medium.com/why-your-git-repo-needs-git-lfs-for-large-files-like-ml-models-e503f3ef42b2?source=rss-eb26b525a52e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e503f3ef42b2</guid>
            <category><![CDATA[git-lfs]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[git-large-files]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[models-in-git]]></category>
            <dc:creator><![CDATA[Amantha Sandupa]]></dc:creator>
            <pubDate>Sun, 28 Sep 2025 17:18:35 GMT</pubDate>
            <atom:updated>2025-09-28T17:18:35.437Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MwNb0VRLXtO6c2_qBfQ8lw.png" /></figure><p>Git excels at version-controlling code, but it struggles with large binary files like ML models, images, or videos. Each update to a large file stores a full copy in the Git history, bloating the repository. My 300 MB model added 300 MB per version, slowing down clones and fetches. Worse, large files can lead to silent failures or corruption during pushes, as I experienced.</p><p>What has happened to me 🥹</p><blockquote><strong>One of our API was working fine until we pushed some new features. The build process completed successfully and it was deployed to the K8S environment without any issues. However, when running, the newly added feature was not working properly. There were no specific logs to troubleshoot, and we had no clue what was going on. Later, we found out that one of our model file which is a large file ( more than 100mb ) was corrupted and was not being fully cloned during the CI/CD pipeline build.</strong></blockquote><p>We realised the issue: Git wasn’t handling the large model file properly in the GitLab pipeline. Pushes seemed to succeed, but the file was incomplete or corrupted.</p><h4>What is Git LFS?</h4><p>Git excels at version-controlling code, but it struggles with large binary files like ML models (<strong>Binary files</strong>), <strong>images</strong>, or <strong>videos</strong>. Each update to a large file stores a full copy in the Git history, bloating the repository. My 300 MB model added 300 MB per version, slowing down clones and fetches. Worse, large files can lead to silent failures or corruption during pushes, as I experienced.</p><p>Git Large File Storage (LFS) is an open-source Git extension designed for large files. Instead of storing the full file in the Git repository, LFS replaces it with a small pointer file , a text file (~100 bytes) containing metadata like a unique ID and file size. The actual file content is stored on a separate LFS server, in this case, GitLab’s LFS storage. This keeps your repository lightweight while enabling version control for large files.</p><p>GitLab fully supports LFS, with no file size limit enforced by default (though practical limits depend on your instance’s storage and configuration). For my microservice, LFS ensured my model was stored reliably, preventing corruption and keeping the repo manageable.</p><h4>How Git LFS Works</h4><ol><li><strong>Tracking Files:</strong> You can specify which files or patterns (e.g., *.bin for model files) should be managed by LFS, or git can identify the LFS and put an <strong>LFS </strong>tag.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*zHifdTgANm0dwpnwtNEebQ.png" /><figcaption><a href="https://docs.gitlab.com/topics/git/lfs/">source — gitlab.com</a></figcaption></figure><p><strong>2. Committing:</strong> You can add a large file as a regular file, and Git LFS intercepts it. It uploads the file to the GitLab LFS server and creates a pointer file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jtP0lb_gLx_UwZ5x" /></figure><p><strong>3. Pushing, Cloning or Checking Out:</strong> During git push, the pointer goes to the GitLab repo, and LFS handles uploading the file to the LFS server. When you clone the repo or switch branches, Git downloads the pointers, and LFS fetches the actual files from the GitLab LFS server, placing them in your working directory.</p><h4>The Solutions for GIT LFS</h4><ul><li><strong>Install Git LFS</strong></li></ul><blockquote>In my case, I added “<strong><em>git lfs install”</em></strong> in my Dockerfile, where the application builds in the CI/CD pipeline, and it worked fine. The large file was cloned properly.</blockquote><ul><li><strong>Track large files</strong></li></ul><pre>git lfs track &quot;your large file&quot;<br>git add .gitattributes # better in handling binary files</pre><ul><li><strong>Migrate existing large files </strong>— this it moves large files from the regular Git storage into LFS, which stores them more efficiently and keeps your repository lightweight</li></ul><pre>git lfs migrate import --include=&quot;your large file&quot;</pre><ul><li>Set up your workflows with git lfs — add git lfs configurations to your Dockerfiles, CI/CD pipelines</li><li>Don&#39;t use large files at all :)</li><li>Monitor Storage: GitLab’s LFS storage counts toward your project’s storage quota. Self-hosted GitLab instances may have custom limits.</li></ul><h4>Conclution</h4><p>Suppose you’re working with large files in GitLab, whether ML models, datasets, or media, Git LFS is essential. Set it up early, configure your CI/CD pipeline to support it, add git lfs to your Dockerfiles, local development environments and save yourself the debugging nightmare I endured. Have you faced similar challenges with large files in GitLab? Share your story in the comments. I’d love to hear how you solved it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e503f3ef42b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data LakeHouse (DataBase vs DataWarehouse vs Data Lake vs Data Lakehouse)]]></title>
            <link>https://amanthaa.medium.com/data-lakehouse-database-vs-datawarehouse-vs-data-lake-vs-data-lakehouse-0e2ed09ad142?source=rss-eb26b525a52e------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e2ed09ad142</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-warehousing]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Amantha Sandupa]]></dc:creator>
            <pubDate>Wed, 12 Feb 2025 09:03:17 GMT</pubDate>
            <atom:updated>2025-02-12T09:12:30.257Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*KqBf85KcVE3vjdQO.jpg" /><figcaption><a href="https://lakehouse.lk/home/lakehouse/">https://lakehouse.lk/home/lakehouse/</a></figcaption></figure><h3>Database</h3><p>Design to capture and record data <br>OLTP — online transactional process <br>Use live, real-time data<br>Data stored in tables with rows and columns <br>Highly detailed<br>Flexible schema<br>Fast for a small amount of data</p><h3><strong>Data Warehouse</strong></h3><p>Designed for the analytical processing of huge amounts of data <br>OLAP — online analytical processing <br>data is refreshed from source systems — stores current and historical data <br>Data is summarized<br>Rigid Schema <br>Fast for large amounts of data<br>Costly to maintain</p><h3>Data Lake</h3><p>Designed to capture raw data <br>Made for large amounts of data <br>Can organize and put into Databases or Data warehouses <br>Cheap to maintain</p><h3>Data LakeHouse</h3><p>We have all sorts of different data coming into our organization from different sources, just like a kitchen has ingredients coming from different suppliers. We need a quick place to dump all different data types in different formats for later use. So, we have data lakes and data warehouses.</p><p>Just like the pantries and freezers in a kitchen, the data in the warehouse is cleaned, organized, and governed and should be trusted for integrity. Developers took the best of both data lakes and data warehouses and combined them into a new technology called the data lakehouse, which has the flexibility and low cost of a data lake and the performance and structure of a data warehouse.</p><blockquote><em>Data Lake + Data Warehouse = Data Lakehouse</em></blockquote><blockquote>Open data lakehouse <br>Open Data Lakehouse specifically uses open-source table formats and standards, enabling access to data from various tools and vendors without vendor lock-in. (Apache Hudi, Delta Lake)</blockquote><p>Three key values are delivered by data warehouse architecture,</p><ol><li>Cost — provides the foundation for users to easily and cost-effectively access, store, manage and unify large amounts of data from different sources and in different formats</li><li>Consume — data lakehouse can be easy to deploy within existing environments, providing users fast access to more data without having long procurement, onboarding or data pipelining and wrangling processes, making it much easier to consume</li><li>Architecture — this can optimize analytics workloads to run where they perform the best and are most cost-efficient. All this is part of an integrated architecture</li></ol><p>A lakehouse should leverage the capabilities across existing data and analytics environments and no need to migrate or rip and replace that environment in order to get started with the lakehouse.</p><p>But what about new data and new workloads? Well, a lakehouse should be the starting point for new data and new workloads, as well as provide a modernization path for existing environments over time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e2ed09ad142" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VS Code Extensions That I Use Everyday]]></title>
            <link>https://amanthaa.medium.com/vs-code-extensions-that-i-use-everyday-7adfbafafff2?source=rss-eb26b525a52e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7adfbafafff2</guid>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[gitlens]]></category>
            <category><![CDATA[vscode-extension]]></category>
            <dc:creator><![CDATA[Amantha Sandupa]]></dc:creator>
            <pubDate>Fri, 05 Jul 2024 15:50:51 GMT</pubDate>
            <atom:updated>2024-07-05T15:52:46.596Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*rruf_pml3YUHBgW9_dofkw.jpeg" /></figure><h3>1. GitLens</h3><p>GitLens is a powerful open-source extension for Visual Studio Code that developed by <a href="https://www.gitkraken.com/">GitKraken</a></p><p>This is the most useful extension for me that i use everyday . Wit this can get powerful insights into my repositories directly in Vs Code.</p><p>One of the most useful features of GitLens is the blame annotation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*HRKYG_r7QI1SWHfilH83UA.png" /></figure><p>url — <a href="https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens">GitLens</a></p><h3>2. Docker Exension</h3><p>The Docker extension makes it easy to build, manage, and deploy containerized applications that is developed by Microsoft . It also provides one-click debugging of Node.js, Python, and .NET inside a container.</p><p>This extension helps me edit and write Dockerfile and docker-compose.yml files. It also assists with completions and syntax for common commands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*KujCdhHRmaUwG7Ua8gb0LA.png" /></figure><p>Docker explorer is tool that offers with this extention and it helps me to manage Docker assets: containers, images, volumes, networks, and container registries. It also supports registries like Github , Dcckerhub and Azure Container Registry .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/0*-8pMpyqITEQVfLOd.gif" /><figcaption>source — <a href="https://code.visualstudio.com/docs/containers/overview">https://code.visualstudio.com/docs/containers/overview</a></figcaption></figure><p>url — <a href="https://code.visualstudio.com/docs/containers/overview">docker extention</a></p><h3>3. Atom One Dark Theme</h3><p>Best rated theme extension in the marketplace, features full ide theming. This improve my code readability by using colors and fonts that reduce eye strain, making it easier to distinguish between different elements of the code.</p><figure><img alt="source — https://marketplace.visualstudio.com/items?itemName=zhuangtongfa.Material-theme" src="https://cdn-images-1.medium.com/max/1024/0*xcZWhHO8WT4Txoqf.png" /><figcaption>source — <a href="https://marketplace.visualstudio.com/items?itemName=zhuangtongfa.Material-theme">https://marketplace.visualstudio.com/items?itemName=zhuangtongfa.Material-theme</a></figcaption></figure><h3>4. Bookmarks</h3><p>I use bookmarks extension because it allows me to bookmark positions in my code. These lines are noted with a blue bookmark icon. Bookmarks can be organized and named to allow for quick reference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/0*aEfuzcQEiZ9u_qBI.png" /><figcaption>bookmark official site</figcaption></figure><h3>5. Todo Tree</h3><p>TODO tree quickly searches the workspace for comment tags like TODO and FIXME, and displays them in a tree view in the activity bar. The view can be dragged out of the activity bar into the explorer pane (or anywhere else you would prefer it to be).</p><p>Since I use VS Code most of the time, I frequently add TODOs. This extension helps me organize my TODOs very well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dF2rquTQgt9hdkR3.png" /><figcaption><a href="https://marketplace.visualstudio.com/items?itemName=Gruntfuggly.todo-tree">https://marketplace.visualstudio.com/items?itemName=Gruntfuggly.todo-tree</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7adfbafafff2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Most Useful K8S Commands for Software Engineers]]></title>
            <link>https://amanthaa.medium.com/most-useful-k8s-commands-for-software-engineers-cf772a6a9d7e?source=rss-eb26b525a52e------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf772a6a9d7e</guid>
            <category><![CDATA[useful-tips]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-trends]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Amantha Sandupa]]></dc:creator>
            <pubDate>Sun, 08 Oct 2023 04:38:36 GMT</pubDate>
            <atom:updated>2025-01-21T05:08:12.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DxnqfQUUzlIjvDouzTYYGA.png" /></figure><p>Kubernetes, aka K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.</p><p>As IT professionals, we often find ourselves dealing with Kubernetes (K8s) but may not be fully equipped to troubleshoot, manage, and work with it effectively. Here are some essential day-to-day K8s commands for non-DevOps individuals to help manage and work within Kubernetes environments.</p><h3>Namespace commands</h3><pre>kubectl config set-context --current --namespace=&lt;namespace-name&gt; #switch to a namespace<br>kubectl get ns                                                    # get all the namespaces<br>kubectl create ns namespace-name                                  # create a namespace<br>kubectl delete ns namespace-name                                  # get all the namespaces<br>kubectl config view | grep namespace                              # check the namespace that logged in</pre><h3>Interacting with pods</h3><pre>kubectl get pods                        # get all pods<br>kubectl get pods -A                     # get all pods in all namespaces<br>kubectl get pods --namespace=name       # get pods in specific namespace<br>kubectl get pods -o wide                # get all pods in the current namespace, with more details<br>kubectl delete pod                      # delete pod<br>kubectl describe pod                    # describe pod<br>kubectl get pod pod-name -o yaml        # get the yaml file of the pod<br>kubectl describe node &lt;node-name&gt;       # describe node<br>kubectl get pods -w                     # watch pod actions realtime<br>kubectl get pods | grep -i &#39;string&#39;     # get pods matching the string<br>kubectl logs -f pod-name                # get logs<br>kubectl run nginx --image=nginx         # start a single pod in mynamespace<br>kubectl exec -it pod-name -- /bin/bash  # create a interacting terminal inside the pod</pre><h3>Node commands</h3><pre>kubectl get nodes                             # get all nodes<br>kubectl get nodes -o wide                     # get all nodes with details<br>kubectl describe nodes                        # describe all nodes <br>kubectl describe node node-name               # describe a specific node</pre><h3>Configmaps</h3><pre>kubectl create cm configmap-name              # create a configmap <br>kubectl get cm                                # get all configmaps<br>kubectl get cm configmap-name -o yaml         # get a configmap as yaml format<br>kubectl edit cm configmap-name                # edit a configmap</pre><h3>Services and Deployments</h3><pre>kubectl get svc                                     # get all services in the namespace<br>kubectl get deployments                             # get all deployments<br>kubectl get deployment dep-name                     # get a particular deployment<br>kubectl scale deployment --replicas=3 depl-name     # scale deployment to 3 pods<br>kubectl delete deployment deployment-name           # delete deplyment <br>kubectl delete svc service-name                     # delete service<br>kubectl rollout restart deployment deployment-name  # restart deployment</pre><pre>kubectl apply -k configs                            # add resorces to cluster<br>kubectl apply -f deployment.yaml                    # create a deployment using resource deployment.yaml</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf772a6a9d7e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>