<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Engineering at NetBook - Medium]]></title>
        <description><![CDATA[Our Engineering team handles all the infrastructure so that Data Scientists can focus on their model building process. We will be writing about how we are building the world-class engineering solution here. - Medium]]></description>
        <link>https://medium.com/engineering-at-netbook?source=rss----dd724e56f6f7---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Engineering at NetBook - Medium</title>
            <link>https://medium.com/engineering-at-netbook?source=rss----dd724e56f6f7---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:16:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/engineering-at-netbook" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How we Implemented Kubernetes pod commit at NetBook]]></title>
            <link>https://medium.com/engineering-at-netbook/how-we-implemented-kubernetes-pod-commit-at-netbook-6638193c223f?source=rss----dd724e56f6f7---4</link>
            <guid isPermaLink="false">https://medium.com/p/6638193c223f</guid>
            <category><![CDATA[container-commit]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[pod-commit]]></category>
            <dc:creator><![CDATA[Ashwani Sharma]]></dc:creator>
            <pubDate>Fri, 29 Jul 2022 05:17:50 GMT</pubDate>
            <atom:updated>2022-07-29T09:28:44.382Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction</h4><p>As we all know, Kubernetes pods are ephemeral, once a pod is stopped or deleted, it loses its state. So, anything installed on the pods is lost on the deletion and you need to re-install it again next time you want to use it again.</p><p>At NetBook, we are building a Kubernetes-backed ML training platform to enable scale for DS. We provide code editors for writing the code and serverless training jobs to run model trainings. However, developers need to install some packages while writing the code which might be missing in their requirements.txt or any package management tool they use.</p><p>So, we want to make sure that the work they do and the packages they install while working need to be persisted. AWS Sagemaker doesn’t support this. Having heard from a lot of Data Scientists about this request, we wanted to solve this.</p><p>As we use docker and Kubernetes for our platform, it was a bit easier for us to crack this nut.</p><p>Almost all the container runtimes provide the feature of committing a container which creates a new image with one extra layer over the existing image with current changes on the container. We can use the same feature to persist the pod changes as well, as a pod also creates a container using a container runtime.</p><h4>The issue we faced at Netbook</h4><p>We at, <a href="https://cloud.netbook.ai/">netbook</a>, provide Workspaces, as we mentioned above are cloud server-backed code editors(jlab or vscode). You can read more about it <a href="https://medium.com/engineering-at-netbook/how-we-are-building-multi-cloud-portable-workspaces-at-netbook-21182a47b8c5">here</a>. The Data Scientists using those workspaces will install various other python packages to help them in their development and they would want those installed tools every time they launch the workspace.</p><p>As we are using Kubernetes pods to implement the workspace and we provide the feature of workspace pause and resume to save on infra cost. Every time we pause a workspace, we remove all infra related to the workspace except the workspace directory which is bound to a volume, so we just keep it for the next resume. After deleting the pod, when we create a new pod for the workspace on resume, we won’t be having anything back which was installed on the previous version of the workspace.</p><p>To overcome this issue, we implemented a service and we call it <em>committer-service.</em></p><h4>Solution</h4><p>As we needed to commit the pod, we had to commit the underlying container of the workspace. So, in committer-service, we came up with a system which will commit the underlying container of the workspace pod and create a new docker image and pushes it to a docker registry whenever we pause the workspace. And, on resuming the workspace we will just use the newer docker image and we will have all changes made in the previous version of the workspace.</p><h4>Impact on end users</h4><p>For our workspaces, the end users are generally <em>Data Science and ML engineers</em>. They create a workspace that comes up with a code editor. With this new feature of pod commit, Data Scientists can use the workspace however they want as if it is their local dev system, install any specific tools they use for development and they shall need to do this just once. They can pause their workspace whenever they are not using it, come back, and resume the workspace and they would still get the same environment as they had with saving on the infra cost.</p><h4>How we implemented pod committing: Technical dive</h4><p><strong>CRI</strong></p><p>As the name of the article suggests, we wanted to implement something which commits the pod whenever we are pausing the workspace. We knew there is a feature of committing the container in docker but as of the latest release, <a href="https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/">kubernetes has moved away from docker</a> and azure aks is already using the <strong>containerd</strong> for CRI, so we explored the similar feature in containerd and yes it also provides the same container commit feature, we tested it using <strong>ctr</strong> <strong>cli .</strong></p><p><strong>Getting the underlying container for a pod</strong></p><p>The next thing to figure out was how do we get the underlying container for a pod. To figure this out, we found that containerd has a separate labelling strategy and labels all the containers, one label was <em>io.kubernetes.pod.name </em>and by using the label as a filter while fetching the containers using from containerd runtime, we got our workspace pod’s container</p><p><strong>Exposing containerd over TCP</strong></p><p>So, now we knew how to get the container for a running pod, now we needed to access the containerd socket of the kubernetes node remotely and for that, we used <em>socat,</em> a Linux utility which is generally available on a Linux node, So we created a kubernetes deployment and respective service which we could access and provide to containerd client to connect to remote containerd server. the <em>socat</em> command we used is:</p><p>sudo socat TCP-LISTEN:{$PORT},reuseaddr,fork UNIX-CONNECT:/run/containerd/containerd.sock</p><p>so, we can now access the containerd remotely as: “socat-service.namespace:{$PORT}” .</p><p><strong>NetBook Workspace</strong></p><p>Now, whenever we pause the workspace, we call the committer service before deleting the respective pod which commits the underlying workspace pod and creates a new image which we push to a docker registry and on resuming the workspace we use the newer image to again set up the workspace.</p><p>This is how we achieved the pod-committing at Netbook.</p><p><strong>End Notes</strong></p><p>We started NetBook with a vision to bring Kubernetes to the model training phase to cut down on model training costs and increase the model training speeds.</p><p>As a part of it, we have functionalities to write code and experiment with multiple hyperparameters.</p><p>If you are a DS team who is tired of AWS Sagemaker or Kubeflow, you can check out NetBook to quickly get started in the AI world with 0 setup.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6638193c223f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-netbook/how-we-implemented-kubernetes-pod-commit-at-netbook-6638193c223f">How we Implemented Kubernetes pod commit at NetBook</a> was originally published in <a href="https://medium.com/engineering-at-netbook">Engineering at NetBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Train Yolo-V5 using Distributed Data Parallel on Multi Node Multi-GPU on AWS in 5 minutes]]></title>
            <link>https://medium.com/engineering-at-netbook/running-yolo-v5-with-ddp-on-aws-8a4f07a77cf?source=rss----dd724e56f6f7---4</link>
            <guid isPermaLink="false">https://medium.com/p/8a4f07a77cf</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[distributeddataparallel]]></category>
            <category><![CDATA[multi-gpu]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Sachin Chandra]]></dc:creator>
            <pubDate>Thu, 09 Jun 2022 17:27:28 GMT</pubDate>
            <atom:updated>2022-06-10T02:17:08.116Z</atom:updated>
            <content:encoded><![CDATA[<p>Setting up Yolo-V5 to run it with a DDP setup on AWS ( Not Sagemaker)</p><p><strong>Introduction</strong></p><p>Yolo has been quite popular amongst all the object detection frameworks and the latest version Yolo-V5 caught our attention.</p><p>We’ve gone through the repo and everything was so well defined and document. But we haven’t found a way to run DDP on multi node using Kubernetes.</p><p>So, we tried setting up and here is a detailed setup guide. This is using solely AWS assets and not managed solutions like Sagemaker. Though Sagemaker provides something like this, it requires you to use Sagemaker specific packaging for your code. Not sure if they directly let you run your code without any Sagemaker SDKs into play. Anyway, let’s go!</p><p>Brace yourself .It is going to be a tedious task. Patiently go through the entire blog and you should pull this off.</p><p>If not, then directly skip to the last section where you can run this <a href="#1825">with 1 click using NetBook.</a></p><p><a href="#3a15"><strong><em>Pre-requisites</em></strong></a></p><p><a href="#3e04"><strong><em>Step 1: Creating Docker Image</em></strong></a></p><p><a href="#801c"><strong><em>Step 2: Setting up EKS clusters</em></strong></a></p><p><a href="#6dac"><strong><em>Step 3: DataSet Setup on EFS</em></strong></a></p><p><a href="#e755"><strong><em>Step 4: Setting up Kubernetes operators</em></strong></a></p><p><a href="#92c6"><strong><em>Step 5: Training and Monitoring</em></strong></a></p><h3><strong>Pre-requisites:</strong></h3><p>Make sure you have the below accesses on your AWS account</p><blockquote>EKS( Managed Kubernetes service)</blockquote><blockquote>EC2( Compute Nodes)</blockquote><blockquote>EFS ( Elastic File system. NFS implementation in AWS)</blockquote><p><strong>Lets get started:</strong></p><p>I assume all of you know how Distributed Data Parallel works for multiple nodes and you have a decent knowledge of Kubernetes.</p><p>First and foremost. Let’s prepare the training docker container. Kubernetes works with only container based architectures. So, let’s set it up.</p><h3><strong><em>Step 1: Creating Docker Image</em></strong></h3><p>You can directly use the YoloV5 <a href="https://hub.docker.com/r/ultralytics/yolov5">official docker</a>. But I prefer setting it up as its not very clear in docker on the last code update it’s been updated on.</p><p>Here is a simple Dockerfile we have created to make sure we have the latest code updated.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47140ec7cdc2100d5bcb3a90a99070a2/href">https://medium.com/media/47140ec7cdc2100d5bcb3a90a99070a2/href</a></iframe><p>You can use the docker file from <a href="https://hub.docker.com/r/sachinnetbook/yolov5-training"><strong>NetBook-Yolo-V5-Docker</strong></a></p><h3><strong><em>Step 2: Setting up EKS clusters</em></strong></h3><p>You can go to AWS Console to do this or use <a href="https://github.com/netbookai/spawner">NetBook’s Spawner</a> to create this in less than 5 minutes.</p><p>NetBook’s Spawner is a multi cloud Kubernetes Cluster creator and manager. It helps you with setting up Kubernetes clusters across GCP, AZURE, AWS.</p><p>Refer to <a href="https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html">this article </a>on how to setup using AWS Console.</p><p><strong><em>ToDo: Publish my version of this with errors and issues found while setup</em></strong></p><h3><strong><em>Step 3: DataSet Setup on EFS</em></strong></h3><p>This is going to be most painful step. Make sure you follow through.</p><p>Looking for alternatives to EFS painful process.</p><p>Exploring if we can use S3 buckets as EFS mounts</p><p>Refer to the <a href="https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html">this article </a>on how to setup EFS through AWS Console</p><p><strong><em>ToDo: Publish my version of this with errors and issues found while setup</em></strong></p><h3><strong><em>Step 4: Setting up Kubernetes operators</em></strong></h3><p>Kubeflow training operators for the rescue to run training jobs on Kubernetes.</p><p>I’ve explored Torch Elastic Operator too, but recent Kubeflow Pytorch operator has this covered.</p><p>To install operators, first let’s connect to our EKS cluster to use Kubectl command.</p><ol><li>Add your AWS access key and secret to AWS config. You should find it in</li></ol><pre>/Users/sachinbonagiri/.aws/credentials</pre><p>2. Check if the creds are working using this</p><pre>aws sts get-caller-identity</pre><p>3. Use this to connect to cluster. This will automatically update your kube config</p><pre>aws eks --region &quot;region&quot; update-kubeconfig --name &quot;cluster_name&quot;</pre><p>4. Check if the connection is working</p><pre>kubectl get nodes<br>Kubectl get pods</pre><p>You should be able to see your AWS node IPs here</p><h4>Installing Kubeflow Training Operator:</h4><p>You can refer to this github repo for <a href="https://github.com/kubeflow/training-operator">Kubeflow Operators</a> and follow the steps. Or simply use the below command to install it</p><pre>kubectl apply -k &quot;github.com/kubeflow/training-operator/manifests/overlays/standalone&quot;</pre><p>This will create a new namespace Kubeflow and start a pod for training operator in this. Check this by running</p><pre>kubectl get pods -n kubeflow</pre><p>You should find the operator pod running</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*qegjR3dp3MC_ZVonHzpszw.png" /></figure><p>There is an alternative to this using <a href="https://pytorch.org/torchx/latest/">TorchX</a>. But haven’t tried it. Looks like its the easiest way to run a job through a CLI.</p><h3><strong><em>Step5 : Starting a Training Job</em></strong></h3><p>For starters, create a separate namespace so that you can manage well.</p><p>To create a namespace run this</p><pre>Kubectl create namespace elastic-job</pre><p>To submit a training job, we better have our training data in a separate storage accessible to multiple nodes. We used EFS for the same.</p><p>Here is the YAML file for provisioning a PV with existing EFS volume</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c6a5098d40cc7b04bb00189df805a93/href">https://medium.com/media/9c6a5098d40cc7b04bb00189df805a93/href</a></iframe><p>Note that as its an AWS EFS, the CSI drivers are relevant only to EKS. If you run on your own nodes, then the setup will be different for PV</p><p>Apply the PV using this command</p><pre>kubectl apply -f efs-sc.yaml -n elastic-job</pre><p>You should see the pv by running this command</p><pre>kubectl get pv</pre><p>You should see the below PV provisioned</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*qMCdGx7VsBT9mwbD6XvXDw.png" /></figure><p>Here is the YAML for provisioning PVCs to enable pods to mount volumes</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7c2e13a82e6fbecbccea0a5729906a3/href">https://medium.com/media/b7c2e13a82e6fbecbccea0a5729906a3/href</a></iframe><p>Note that its a ReadWriteMany as we need multiple pods in multiple nodes to read from this volume. We will use this volume for the weight file saving too so that when ever training needs to be resumed, it can read the checkpoint file from here and resume.</p><p>To apply the yaml run the below command</p><pre>kubectl apply -f pvc_claim.yaml -n elastic-job</pre><p><strong>Note</strong>: PVs are not namespace bound. So we didn’t mention namespace in the while applying above PV. But PVCs are. So mention the namespace</p><p>You should see the PVCs by running this command</p><pre>kubectl get pvc -n elastic-job</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*MSPScHoFdMZpRK1tBTDDGA.png" /></figure><p><strong>Note</strong>: Now our PVCs are claimed from the EFS PV we provisioned and you can see the access modes as RWX. ( ReadWriteMany). The one we need.</p><p>Now finally, start the training job</p><p>Here is the YAML file we’ve used to start training</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12cb6ff10282ef623ce10a8aca764e76/href">https://medium.com/media/12cb6ff10282ef623ce10a8aca764e76/href</a></iframe><p>Now if there are no errors, you should see the pods running in the namespace</p><pre>Kubectl get pods -n elastic-job -o wide</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ohqZSAJM8uTP3l-pP5s2WQ.png" /></figure><p>You can see 3 workers running on 3 different nodes.</p><p>Bingo, here you go, you scheduled a distributed training job on 3 different nodes.</p><p>Here are some observations:</p><p>I was using g4dn.xlarge instance of AWS-EC2</p><ol><li>DDP on 2 nodes: Each epoch took around 12 mins 50 seconds. 3.2 iterations/second</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9DlxJqwX8bzmtyz_XSHxg.png" /></figure><ol><li>DDP on 1 node: Each epoch took around 24mins 37 seconds. 1.68 iterations/second</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BdQZV-pJ3QZA9zOGzEaAxg.png" /></figure><p>That&#39;s close to 2x training speeds.</p><p>You can also add SPOT nodes to the training. But make sure that you have an on-demand node at least for the master-worker to not fail.</p><p>The elasticity is managed by Pytorch. So, the training should resume even when the other node goes down.</p><h3>Server-less Model Training in 1-Click on NetBook</h3><p>As we’ve seen how painful it is for Data Science teams to set up infrastructure for model training on Kubernetes. We at <a href="https://cloud.netbook.ai/">NetBook</a> are doing the hard work for you to run the above-mentioned things with just a click of a button.</p><p>We currently support single-node training which is now live on the platform. But this <strong><em>June 22nd</em></strong>, we are coming up with a special entry on the platform for folks who want to train Yolo-V5 with your own data on <a href="https://cloud.netbook.ai/">NetBook</a> with a 0 Infra setup.</p><p><strong><em>You can </em></strong><a href="https://mc2zpchqy0j.typeform.com/to/NNcT8idU"><strong><em>signup here</em></strong></a><strong><em> to book your slot. Slots are limited as training jobsare going to be expensive.</em></strong></p><p>If you face any issues in setup, do let me know, and happy to help. You can find me on <a href="https://www.linkedin.com/in/sachin-chandra-043a0178/">LinkedIn</a>.</p><p>If you want to collaborate, mail me at sachin@netbook.ai</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a4f07a77cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-netbook/running-yolo-v5-with-ddp-on-aws-8a4f07a77cf">Train Yolo-V5 using Distributed Data Parallel on Multi Node Multi-GPU on AWS in 5 minutes</a> was originally published in <a href="https://medium.com/engineering-at-netbook">Engineering at NetBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we implemented Pod Logging at NetBook]]></title>
            <link>https://medium.com/engineering-at-netbook/how-we-implemented-pod-logging-at-netbook-29608fffe95a?source=rss----dd724e56f6f7---4</link>
            <guid isPermaLink="false">https://medium.com/p/29608fffe95a</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Mano Sriram]]></dc:creator>
            <pubDate>Mon, 23 May 2022 15:49:29 GMT</pubDate>
            <atom:updated>2022-05-23T15:49:28.950Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_6-VqzKqv3IKf3eEinRrQ.png" /></figure><p>Each Workspace or Experiment is deployed as a Pod in a Cluster. Usually, the user will need to see the workspace or the experiment logs after deployment.</p><blockquote>A <strong>Workspace</strong> is simply an editor (JupyterLab or VsCode) on which users can work.</blockquote><blockquote><a href="http://netbook.ai/"><strong>NetBook</strong></a> enables Data Scientists to run 10s of model training jobs in parallel to cut down the training times. All these training jobs are clubbed under <strong>Experiments</strong> section on NetBook.</blockquote><p>Starting with this feature, we thought of using Fluentd to collect Pod logs and store them in a Kafka queue. The approach was a bit complicated but it might work. Working on it parallelly, we were also looking for simpler solutions to make logs reach the end user.</p><p>That’s when we found the Kubernetes <strong>GetLogs</strong> API to get logs of a Pod.</p><p>Using this API, we could easily get logs of a specific pod or all pods in a namespace. So, we dumped our Kafka idea and immediately started working on this.</p><p>The idea was that the <strong>Placement Service</strong> would grab logs from the Pod and the <strong>Backend</strong> would stream it to the client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8BD2YEx7P1IWHY9GmtMH6w.png" /><figcaption>pod logs flow</figcaption></figure><p>There were some hurdles that we needed to think through.</p><ol><li>How does the user get a live feed of logs given that we don’t have the bandwidth to set up web sockets.</li><li>How do we handle logs for both workspaces and experiments, the difference being experiments will have multiple pods.</li></ol><h4>Gin’s Stream API and Long Polling</h4><p>The alternative to web sockets that we knew was long-polling. The client could open a connection for an extended period of time and the backend would stream the logs as it received. It was a sensible thing to do, and we did the same.</p><p>There were few channels needed to achieve log streaming.</p><ol><li><strong>Log </strong>Channel to receive logs from the placement service.</li><li><strong>Done</strong> channel which placement service (or) the client triggers to stop streaming.</li><li><strong>Timeout</strong> channel, set to 15 minutes.</li><li><strong>Error</strong> channel to handle errors.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*96uHT3-sWfC9bm_xcLHqYw.png" /><figcaption>using go channels to receive logs, errors, and done.</figcaption></figure><p>In Placement Service, <strong>ReadLogs</strong> is spawned as a go-routine. The function does the following.</p><ul><li>Read logs from the Pod</li><li>Check if the given context is canceled, break if so.</li><li>Handle EOF (end-of-file) of the log.</li><li>Wrap the log from Pod into the response struct and stream it back to the Backend.</li><li>Close the Kubernetes Pod Log Stream channel when done.</li></ul><p>Workspace’s Pod name is stored in DB which could be easily used to get its logs. With Experiments, we have a namespace. So, we just handled this case by sending logs of all pods if a namespace is specified, else just send the given pod’s logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hOc5_4-cRQvIe8MOioaR8A.png" /><figcaption>pod logging for workspaces and experiments</figcaption></figure><h4>Needed Improvements</h4><p>The immediate next thing is to store the logs in case the pod fails or is destroyed. For this case, we can use Kafka to store the logs of a pod. We can create a topic for each pod with the topic name as the pod’s name.</p><h4>Open Source at NetBook</h4><p>We at NetBook have started open sourcing some of our micro-services. Here’s the <a href="https://github.com/netbookai"><strong>link</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29608fffe95a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-netbook/how-we-implemented-pod-logging-at-netbook-29608fffe95a">How we implemented Pod Logging at NetBook</a> was originally published in <a href="https://medium.com/engineering-at-netbook">Engineering at NetBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we use Helm Charts to handle Workspaces at NetBook.]]></title>
            <link>https://medium.com/engineering-at-netbook/how-we-use-helm-charts-to-handle-workspaces-at-netbook-fe5b81d1988e?source=rss----dd724e56f6f7---4</link>
            <guid isPermaLink="false">https://medium.com/p/fe5b81d1988e</guid>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[Mano Sriram]]></dc:creator>
            <pubDate>Sun, 24 Apr 2022 06:28:37 GMT</pubDate>
            <atom:updated>2022-04-24T06:28:37.090Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*vi6Wx5P9QQf54PyFVsU4aA.png" /><figcaption>Kubernetes + Helm</figcaption></figure><p>As you know <a href="https://www.netbook.ai/"><strong>NetBook</strong></a> helps teams handle multi-cloud DS development with ease and, we do it through portable workspaces.</p><p>To deploy a single workspace, it needed many arguments to create a Kubernetes Deployment which in turn creates services and pods. For example, a workspace needs to have a Port, the name of the deployment, the environment image to build the workspace on, and many more. To put in arguments every single time manually was a tedious task when automation tools like Helm were available.</p><p>That&#39;s what the Helm charts do. They automate the Kubernetes deployment. We can have whatever config we want to and helm deploys it with a single command.</p><p>We could use the Kubernetes’s apply command, but it could become a task in the long run. Also, the Kubernetes <strong>apply</strong> doesn’t have versioning support whereas Helm does.</p><p>As of now, NetBook supports 2 types of workspaces: <strong>Jupyter Lab</strong> and <strong>VScode</strong>.</p><p>NetBook uses separate charts for each type of workspace. It is possible to have a single chart for both of them but separating the charts felt like a good thing to do.</p><p>NetBook doesn’t store charts locally since it could create a dependency. We host our charts on Cloudsmith. The <strong>Placement</strong> microservice pulls the helm chart (Jlab or VScode) before deploying them onto the node (instance).</p><h4>Gitlab vs Cloudsmith</h4><p>When we were starting with Helm Charts, we started with Gitlab’s Package Registry. Soon enough, we started facing problems where the charts weren’t updating.</p><p>Even after pushing the latest version, Gitlab used to send the old versions. It was frustrating which made us switch to <a href="https://cloudsmith.com/"><strong>Cloudsmith</strong></a>. We haven’t gone back since.</p><h4>Helm Charts for AWS and Azure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gfCJzKqfm_gQNVVFhUwOXA.png" /><figcaption>AWS or Azure Helm if-else</figcaption></figure><p>Since we have separated charts for Jlab and VScode, it might be sensible to separate charts for these providers too. But if we do so, the number of charts becomes higher than we can handle at this point in time.</p><p>Also, there are only a few places where changes are needed. Helm has if-else support which we made use of to know if the deployment is being done on AWS or Azure</p><p>The Placement Service sends the needed arguments to the Helm Chart. It also sends a flag denoting the provider which we picked up in the Helm Chart to differentiate providers.</p><h4>Versioning Charts</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eIf8DyQJzZZDWZAf_wHYZQ.png" /><figcaption>Helm Versioning</figcaption></figure><p>Helm charts support versioning. This means that every chart can have a version and when deploying a workspace, we could pick the version we need.</p><p>The reason this is useful is that let’s say developers are testing a new feature. They host the chart as <strong>0.2.3</strong> and the production uses the chart <strong>0.2.2</strong>. The version isn’t taken as any argument from any service but stored as an environment variable. The developers can use their version to test the latest features.</p><p>We can also pull the latest version by giving <strong>0.2.</strong>*, which is really useful.</p><p>This is how Helm charts are being used at NetBook. Helm charts sure save a lot of time and are easily manageable.</p><p>Resources:<br><a href="https://helm.sh/">https://helm.sh/</a><br><a href="https://helm.sh/docs/">https://helm.sh/docs/</a><br><a href="https://jfrog.com/blog/10-helm-tutorials-to-start-your-kubernetes-journey/">https://jfrog.com/blog/10-helm-tutorials-to-start-your-kubernetes-journey/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe5b81d1988e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-netbook/how-we-use-helm-charts-to-handle-workspaces-at-netbook-fe5b81d1988e">How we use Helm Charts to handle Workspaces at NetBook.</a> was originally published in <a href="https://medium.com/engineering-at-netbook">Engineering at NetBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we are building Multi-Cloud Portable Workspaces at NetBook]]></title>
            <link>https://medium.com/engineering-at-netbook/how-we-are-building-multi-cloud-portable-workspaces-at-netbook-21182a47b8c5?source=rss----dd724e56f6f7---4</link>
            <guid isPermaLink="false">https://medium.com/p/21182a47b8c5</guid>
            <category><![CDATA[multi-cloud]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Mano Sriram]]></dc:creator>
            <pubDate>Sun, 20 Mar 2022 17:53:11 GMT</pubDate>
            <atom:updated>2022-03-21T07:13:31.140Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t7GrGl-cx3isB132YsNaeg.jpeg" /></figure><p>At NetBook we aim to make infrastructure usage easier for Data Scientists on any cloud provider. To make it happen, the first step is to create work environments for them.</p><p>We built Workspaces to tackle this problem. A <strong>Workspace</strong> is simply an editor (JupyterLab or VsCode) on which users can work.</p><p>An Environment is a collection of libraries built into a docker image. There are few default environments and the user can create images with needed libraries. The environment can be created from:</p><ul><li>Docker Hub</li><li>Github Repo</li><li>Dockerfile</li><li>Custom list of libraries selected by the user</li></ul><p><strong>Workspaces</strong> are built on top of an environment so that the libraries can be used in the workspace. They can be spawned with different options ranging from the size of the disk to the region where the workspace needs to be spawned.</p><p>Multiple services are responsible for making this work. Microservices are used to manage different jobs and API requests are handled by an API gateway that interacts with the microservices.</p><p><strong>gRPC</strong> and <strong>Protocol Buffers</strong> are used to communicate with microservices. Golang is used to create most services along with Python. Netbook mostly depends on <strong>Kubernetes</strong> and <strong>Docker</strong>, using them in every single microservice.</p><p>Multiple services are handling multiple jobs, but there are 2 of them responsible for creating a workspace. Every service is a Kubernetes deployment. So each service will have its Pod, Service, PV, and PVCs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLiJf-LmjcBXKv3Qd0VyVg.png" /></figure><p>Helm Charts are used to make automated deployments. All the user’s preferences are given to the helm chart as arguments and Helm installs the respective chart.</p><p><strong>To create a workspace, there are a few dependencies:</strong><br> a. Create EC2 instances on which the workspace runs.<br> b. Create Persistent volumes and Persistent volume claims.<br> c. Deploy the workspace with different options the user provides.</p><h3>Spawner Service</h3><p>It is responsible for finding clusters and creating EC2 instances in clusters that match requirements.</p><p>Since the service runs on a Pod, if it crashes; the data in it will be lost. To solve this problem we create a Persistent Volume (PV) for every workspace so that data is persisted. The user can claim storage (PVC) depending on the workspace.</p><p>Another problem when using these providers is that they all have a specific limit to each of their services which we might not notice until we hit the limit. One needs to keep an eye on the limits to acknowledge we’re not in the danger zone.</p><h3>Placement Service</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*parSdGUeFAIA7yv6-Wee1g.png" /></figure><p>This service deploys the workspace onto an EC2 node. Since we have a node, we need to identify it so that we know the exact node to deploy on. By default, we don’t have any unique identifier for the node, so when creating the node we add a label to identify the node.</p><p>The placement service triggers the helm install which installs the chart as well as deploys the workspace on the node.</p><p>In the helm chart, along with all services; we also add Load Balancer service to expose the URL where the workspace is deployed.</p><p>The UI has to show the status of the workspace creation. Hence, we need an API separately for service status. The problem with this is that even after the workspace is deployed, it takes some time for the pod to start running. The Pod won’t be accessible until it runs. Hence, we check for the Pod’s status to be “Running” to make sure it’s running and accessible.</p><p>Before deploying the workspace, we have to be sure that the node exists and poll until the node is created and then call the placement service to deploy the workspace.</p><h3>API Gateway</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SsfESjilJ8PNL70NGsVBhA.png" /></figure><p>Since support for gRPC is not straightforward, an API gateway is a nice solution. It is a traditional REST API built with Golang. Some of the microservices are triggered on Go Routines since it might take a lot of time and the browser needs a response immediately.</p><h3>Protocol Buffers Vendoring</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HYVpOcxU1PVbmrR6J6dGAg.png" /></figure><p>Since microservice don’t depend on each other, we need to have a vendoring system for proto files so that all services have the updated copy of the proto file. We need vendoring since we getting the proto file by calling the service is being dependent on that service.</p><p>We use the <a href="https://github.com/NetBook-ai/protovend"><strong>protovend</strong></a> tool to handle proto files from different services.</p><p>This tool helps each service to have a copy of the other service’s proto file. This helps when calling other services since we already know what the request is going to look like.</p><h3>Conclusion</h3><p>Netbook currently supports AWS, but we’re starting to bring Azure and GCP support for workspaces which will be a great addition to what we’re providing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21182a47b8c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-netbook/how-we-are-building-multi-cloud-portable-workspaces-at-netbook-21182a47b8c5">How we are building Multi-Cloud Portable Workspaces at NetBook</a> was originally published in <a href="https://medium.com/engineering-at-netbook">Engineering at NetBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Cloud Costs for Deep Learning Trainings]]></title>
            <link>https://medium.com/engineering-at-netbook/optimizing-cloud-costs-for-deep-learning-trainings-8e3556748aca?source=rss----dd724e56f6f7---4</link>
            <guid isPermaLink="false">https://medium.com/p/8e3556748aca</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Shruti Sharma]]></dc:creator>
            <pubDate>Thu, 17 Mar 2022 10:44:20 GMT</pubDate>
            <atom:updated>2022-03-17T10:44:20.467Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*bGOY7RPj2kJeJMDpzvgKSg.jpeg" /><figcaption>Cloud Computing is becoming increasingly popular day-by-day (Image Source: <a href="https://in.pinterest.com/">Pinterest</a>)</figcaption></figure><p>Selecting a cloud instance configuration that suits your workload and at the same time, taking care of your budget is an overdo that data scientists face when they set foot on conducting experiments. In fact, how do you choose which cloud provider works best for your business?</p><p>The major cloud providers —<strong> </strong><a href="https://aws.amazon.com/"><strong>AWS</strong></a><strong>, </strong><a href="https://azure.microsoft.com/en-us/"><strong>Azure</strong></a>,<strong> and </strong><a href="https://cloud.google.com/"><strong>GCP</strong></a> — look similar on the surface offering -</p><ul><li><em>Huge </em><strong><em>compute </em></strong><em>power</em></li><li><em>Distributed </em><strong><em>resources</em></strong></li><li><em>Large </em><strong><em>storage</em></strong><em> spaces</em></li></ul><p>along with facilities like <strong><em>autoscaling </em></strong>and<em> </em><strong><em>flexibility </em></strong>that reduce the burden of a developer. But when you dig deeper, you will find stark differences in their pricing that can send your company’s bills to a toss. Deploying AI applications in the real world requires extensive training and testing to reach considerable accuracy in order to deliver trustworthy results. Therefore, before going on to choose one cloud infrastructure, you will need to have a clear understanding of the <em>dataset, framework,</em> and <em>workload</em> that you will need. <strong>Compute</strong> will always be your highest cost driver. Minor changes in per-hour pricing and switching to alternate instances can save hundreds of dollars.</p><blockquote>We at <a href="https://www.netbook.ai/">NetBook</a> identified this problem and tried to create a simple solution that can save you hours of researching, documentation scraping and then selecting an architecture to go on with a Data Science task. We compiled open-source data of multiple GPU/CPU instances on different frameworks and datasets, clubbed it with the per-hour pricing information of each cloud provider and bundled it all into a Python package that will provide you with alternate cheaper and/or faster cloud instances for the task at hand.</blockquote><p>NVIDIA houses <a href="https://developer.nvidia.com/deep-learning-performance-training-inference">training performance data</a> on NVIDIA <strong>A100, A40, T4, V100</strong> (and many more) GPU instances with benchmarking experiments conducted on numerous networks and datasets on major frameworks like <em>PyTorch, MXNet</em>, and <em>Tensorflow. </em>We combined this extensive data with the hourly pricing information of 4 regions:</p><ul><li><em>US-East-1</em></li><li><em>US-East-2</em></li><li><em>US-West-2</em></li><li><em>AP-South-1</em></li></ul><p><a href="https://developer.nvidia.com/deep-learning-performance-training-inference">NVIDIA Data Center Deep Learning Product Performance</a></p><p>While collecting the pricing data, we observed a huge disparity in cloud costs region-wise and also across different cloud providers. This affirmed our hypothesis on how choosing the right instance is of primary importance for your business alignment. We tried to create a<strong><em> multi-card instance suggestor.</em></strong></p><p>For instance, the one that could convert <em>A100s </em>into cheaper <em>K80s, </em>thus trading your time with cost. We provide you with an option to choose the region that has the lowest costs based on your experiment specifications.</p><p>When data scientist takes hold of a job, they usually are not aware of the infrastructure design to house their experiment. Therefore, another version of our <em>Instance Suggestor </em>takes into account your preferred framework and dataset and thereby suggests you the GPU instances which you can utilize to run your deep learning experiment. We also provide you an option to choose a fast or a cheap alternative from the suggested instances.</p><p>We were able to develop an Instance Suggestor that takes in your specifications about the region, dataset, framework, and cloud provider and informs you about alternate instances on either the same provider and/or a different provider that is cheaper or faster, depending upon your bandwidth and budget.</p><p>For now, the product suggests instances based on <strong><em>single-node training costs and training times</em></strong>. Although, in the future, we plan to extend it to</p><ul><li>Multi-node training convergence statistics.</li><li>Incorporating more datasets and regions for holistic instance suggestions.</li><li>Running the experiment for a few epochs and suggestions based on the estimated time of convergence</li><li>Incorporating instance utilization levels on different experiments to refine the suggestions.</li><li>Dynamic updating of pricing data</li></ul><p>The entire approach has been published as an open-source package on</p><blockquote><strong>Github</strong> :</blockquote><p><a href="https://github.com/NetBook-ai/Instance-suggestor">GitHub - NetBook-ai/Instance-suggestor: Cloud Instance suggestor for Deep Learning Training jobs</a></p><blockquote><strong>PyPi</strong> :</blockquote><p><a href="https://pypi.org/project/InstanceSuggestor/">InstanceSuggestor</a></p><p>We invite collaborations and suggestions on improving the current state of the module. Feel free to raise an issue on Github or connect with me on <a href="https://www.linkedin.com/in/shrutis10/">LinkedIn</a>. You can also keep a track of updates on our product here: <a href="https://www.linkedin.com/company/netbook-ml/">NetBook</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e3556748aca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-at-netbook/optimizing-cloud-costs-for-deep-learning-trainings-8e3556748aca">Optimizing Cloud Costs for Deep Learning Trainings</a> was originally published in <a href="https://medium.com/engineering-at-netbook">Engineering at NetBook</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>