<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rajat Sharma on Medium]]></title>
        <description><![CDATA[Stories by Rajat Sharma on Medium]]></description>
        <link>https://medium.com/@rajatsharma_33357?source=rss-cac8a492636------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EahaGlm-IDumZlniLQmnwQ.jpeg</url>
            <title>Stories by Rajat Sharma on Medium</title>
            <link>https://medium.com/@rajatsharma_33357?source=rss-cac8a492636------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:33:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rajatsharma_33357/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fine-tuning LLMs using LoRA]]></title>
            <link>https://medium.com/@rajatsharma_33357/fine-tuning-llama-using-lora-fb3f48a557d5?source=rss-cac8a492636------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb3f48a557d5</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[fine-tuning-transformer]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Rajat Sharma]]></dc:creator>
            <pubDate>Fri, 15 Sep 2023 08:58:02 GMT</pubDate>
            <atom:updated>2023-09-15T09:27:31.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a1tvGxCiW4VNsx_r.png" /><figcaption>Image From: <a href="https://huggingface.co/blog/evaluating-mmlu-leaderboard">https://huggingface.co/blog/evaluating-mmlu-leaderboard</a></figcaption></figure><p>Since the beginning of the year, LLMs have been making waves, much like computer vision did a decade ago. They’ve been put to use in a wide range of language-related tasks, from summarization and sentiment analysis to translation and various other specialized applications.</p><p>You’ve probably heard of ChatGPT, which, in my opinion, is the top-notch language model (BARD doesn’t even come close). It can handle all the tasks we mentioned earlier, but there’s a caveat: using ChatGPT or any OpenAI models may involve sharing your data with ChatGPT or Microsoft idk, so it’s something to consider 😉.</p><p>A few weeks ago, Meta released the Llama2 models, which are open source and can be utilized for commercial purposes. You can even run them on your in-house infrastructure. So, here in this blog we are going to fine-tune a Llama 7B model on a summarization dataset which you can extend to your domain specific examples.</p><p>First of all install all the dependencies using pip</p><pre>transformers<br>bitsandbytes<br>sentencepiece<br>transformers[sentencepiece]<br>accelerate<br>bitsandbytes<br>datasets<br>peft<br>trl<br>py7zr # for the samsum data</pre><p>Now, we import our dependencies at the top 😃</p><pre>import torch<br>import transformers<br>import argparse<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from datasets import load_dataset<br>from peft import (<br>        get_peft_model, <br>        prepare_model_for_kbit_training, <br>        LoraConfig<br>    )<br>from trl import SFTTrainer</pre><p>Now, to load the LLAMA2 model from Hugging Face, you can take advantage of the fact that Hugging Face has already converted LLAMA2 to their format. Loading it in 8-bit reduces GPU memory consumption from 13 GB to 8 GB, making it quite efficient. You can even push it further to 4-bit using bit and bytes configuration. However, I experienced that the 4-bit configuration didn’t yield the best results after fine-tuning, possibly requiring some hyperparameter adjustments. If you happen to have any insights or suggestions for improvement, feel free to share! 😺</p><p><strong>Please note </strong>that you’ll need a Hugging Face token to access and fetch the model.</p><pre>model_name = &quot;meta-llama/Llama-2-7b-hf&quot;<br>model = AutoModelForCausalLM.from_pretrained(model_name,<br>                                            load_in_8bit=True,<br>                                            device_map=&quot;auto&quot;<br>                                            )<br>tokenizer = AutoTokenizer.from_pretrained(model_name)</pre><p>We’ll now load the dataset (<a href="https://huggingface.co/datasets/samsum">https://huggingface.co/datasets/samsum</a>), and print a row from it and how it looks like</p><pre>data = load_dataset(&quot;samsum&quot;)<br>data_train, data_test, data_val = data[&quot;train&quot;], data[&quot;test&quot;], data[&quot;validation&quot;]<br><br>print(data_train, data_test, data_val)<br><br># example<br>data_train[0]<br># output<br>#{&#39;id&#39;: &#39;13818513&#39;,<br># &#39;dialogue&#39;: &quot;Amanda: I baked  cookies. Do you want some?\r\nJerry: Sure!\r\nAmanda: I&#39;ll bring you tomorrow :-)&quot;,<br># &#39;summary&#39;: &#39;Amanda baked cookies and will bring Jerry some tomorrow.&#39;}</pre><p>We have the data in hand, and we can definitely give it a shot with the base model. Let’s see how the base model performs! First of all we’ll create a generate prompt function that basically creates a prompt to be fed to tokenizer and we’ll also use the <strong>generate_prompt </strong>during the training and inference. You’ll notice that we add “&lt;/s&gt;” (llama2 EOS token) as EOS token. Adding this the model will know when to stop it self.</p><pre>def generate_prompt(dialogue, summary=None, eos_token=&quot;&lt;/s&gt;&quot;):<br>  instruction = &quot;Summarize the following:\n&quot;<br>  input = f&quot;{dialogue}\n&quot;<br>  summary = f&quot;Summary: {summary + &#39; &#39; + eos_token if summary else &#39;&#39;} &quot;<br>  prompt = (&quot; &quot;).join([instruction, input, summary])<br>  return prompt<br><br>print(generate_prompt(data_train[0][&quot;dialogue&quot;], data_train[0][&quot;summary&quot;]))<br># Summarize the following:<br># Amanda: I baked  cookies. Do you want some?<br># Jerry: Sure!<br># Amanda: I&#39;ll bring you tomorrow :-)<br>#  Summary: Amanda baked cookies and will bring Jerry some tomorrow.</pre><p>Now let’s feed the some random conversation data to the model and see its response. As you can see its just a nuisance because the base model are trained on predicting the next probable token liek a sequence model. So it basically fails on the summarization.</p><pre>input_prompt = generate_prompt(data_train[50][&quot;dialogue&quot;])<br>input_tokens = tokenizer(input_prompt, return_tensors=&quot;pt&quot;)[&quot;input_ids&quot;].to(&quot;cuda&quot;)<br>with torch.cuda.amp.autocast():<br>  generation_output = model.generate(<br>      input_ids=input_tokens,<br>      max_new_tokens=1000,<br>      do_sample=True,<br>      top_k=10,<br>      top_p=0.9,<br>      temperature=0.3,<br>      repetition_penalty=1.15,<br>      num_return_sequences=1,<br>      eos_token_id=tokenizer.eos_token_id,<br>    )<br>op = tokenizer.decode(generation_output[0], skip_special_tokens=True)<br>print(op)<br># Summarize the following:<br>#  Pitt: Hey Teddy! Have you received my message?<br># Teddy: No. An email?<br># Pitt: No. On the FB messenger.<br># Teddy: Let me check.<br># Teddy: Yeah. Ta!<br># Summary: 1) What is the purpose of this conversation? 2) Who initiates it and who responds to it? 3) How do they communicate with each other? 4) What does the speaker think about his friend’s response?</pre><p>Now, let’s dive into the main task: Fine-Tuning. There are several methods available for fine-tuning LLMs, such as LoRA, QLoRA (Quantized LoRA), and DeepSpeed. We’re going with LoRA, which you can read more about here: <a href="https://huggingface.co/docs/peft/conceptual_guides/lora">LoRA Documentation</a>. LoRA stands out for its efficiency, as it optimizes only specific modules within the model rather than the entire model. Since we already have the model loaded, lets proceed further and create a LoRA config and prepare the model for the training. You can read more about the parameters in the LoRA docs.</p><pre>lora_config = LoraConfig(<br>        r=8,<br>        lora_alpha=8,<br>        lora_dropout=0.1,<br>        target_modules=[&quot;q_proj&quot;,&quot;k_proj&quot;,&quot;v_proj&quot;,&quot;o_proj&quot;],<br>        bias=&quot;none&quot;,<br>        task_type=&quot;CAUSAL_LM&quot;,<br>    )</pre><p>You’ll notice that <strong>target_modules </strong>list training modules which are specific to Llama2 only. Maybe if you want to fine-tune <strong>google flan-tf-xl</strong>, it would be different. You can see the attention blocks by just printing the model and add accordingly to the training blocks.</p><pre>print(model)<br># LlamaForCausalLM(<br>#  (model): LlamaModel(<br>#     (embed_tokens): Embedding(32000, 4096)<br>#     (layers): ModuleList(<br>#      (0-31): 32 x LlamaDecoderLayer(<br>#        (self_attn): LlamaAttention(<br>#          (q_proj): Linear8bitLt(in_features=4096, out_features=4096, bias=False)<br>#          (k_proj): Linear8bitLt(in_features=4096, out_features=4096, bias=False)<br>#          (v_proj): Linear8bitLt(in_features=4096, out_features=4096, bias=False)<br>#          (o_proj): Linear8bitLt(in_features=4096, out_features=4096, bias=False)<br>#          (rotary_emb): LlamaRotaryEmbedding()<br>#        )<br>#        (mlp): LlamaMLP(<br>#          (gate_proj): Linear8bitLt(in_features=4096, out_features=11008, bias=False)<br>#          (up_proj): Linear8bitLt(in_features=4096, out_features=11008, bias=False)<br>#          (down_proj): Linear8bitLt(in_features=11008, out_features=4096, bias=False)<br>#          (act_fn): SiLUActivation()<br>#        )<br>#        (input_layernorm): LlamaRMSNorm()<br>#        (post_attention_layernorm): LlamaRMSNorm()<br>#      )<br>#    )<br>#    (norm): LlamaRMSNorm()<br>#  )<br>#  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)<br>#)</pre><p>Also we are doing some pre settings. We add a PAD token and resize the model embeddings to the new size.</p><pre># this should be set for finutning and batched inference<br>tokenizer.add_special_tokens({&quot;pad_token&quot;: &quot;&lt;PAD&gt;&quot;})<br>model.resize_token_embeddings(len(tokenizer))</pre><p>Now we’ll prepare the model to LoRA. First, we convert the existing model to 8-bit quantization and then apply the LoRA config to it.</p><pre># Loading in 8 bit ...&quot;<br>model = prepare_model_for_kbit_training(model)<br>model = get_peft_model(model, lora_config)</pre><p>If you print the total trainable parameters it would be 0.089 % or 6291456. Quite less if you compare it with the actual which are 262410240.</p><p>Excellent! With our model now prepared for fine-tuning, it’s time to set up the training arguments. We’ll fine-tune the model for a total of 50 steps.</p><pre>output_dir = &quot;cp&quot;<br>per_device_train_batch_size = 4<br>gradient_accumulation_steps = 4<br>per_device_eval_batch_size = 4<br>eval_accumulation_steps = 4<br>optim = &quot;paged_adamw_32bit&quot;<br>save_steps = 10<br>logging_steps = 10<br>learning_rate = 5e-4<br>max_grad_norm = 0.3<br>max_steps = 50<br>warmup_ratio = 0.03<br>evaluation_strategy=&quot;steps&quot;<br>lr_scheduler_type = &quot;constant&quot;<br><br>training_args = transformers.TrainingArguments(<br>            output_dir=output_dir,<br>            per_device_train_batch_size=per_device_train_batch_size,<br>            gradient_accumulation_steps=gradient_accumulation_steps,<br>            optim=optim,<br>            evaluation_strategy=evaluation_strategy,<br>            save_steps=save_steps,<br>            learning_rate=learning_rate,<br>            logging_steps=logging_steps,<br>            max_grad_norm=max_grad_norm,<br>            max_steps=max_steps,<br>            warmup_ratio=warmup_ratio,<br>            group_by_length=True,<br>            lr_scheduler_type=lr_scheduler_type,<br>            ddp_find_unused_parameters=False,<br>            eval_accumulation_steps=eval_accumulation_steps,<br>            per_device_eval_batch_size=per_device_eval_batch_size,<br>        )</pre><p>Final step, we are using <a href="https://huggingface.co/docs/trl/sft_trainer">SFTTrainer</a> to fine-tune the model. And, that’s it you are done with the writing stuff.</p><pre>def formatting_func(prompt):<br>  output = []<br><br>  for d, s in zip(prompt[&quot;dialogue&quot;], prompt[&quot;summary&quot;]):<br>    op = generate_prompt(d, s)<br>    output.append(op)<br><br>  return output<br><br><br>trainer = SFTTrainer(<br>    model=model,<br>    train_dataset=data_train,<br>    eval_dataset=data_val,<br>    peft_config=lora_config,<br>    formatting_func=formatting_func,<br>    max_seq_length=1024,<br>    tokenizer=tokenizer,<br>    args=training_args<br>)<br><br># We will also pre-process the model by upcasting the layer norms in float 32 for more stable training<br>for name, module in trainer.model.named_modules():<br>    if &quot;norm&quot; in name:<br>        module = module.to(torch.float32)<br><br>trainer.train()<br>trainer.save_model(f&quot;{output_dir}/final&quot;)<br><br># Step Training Loss Validation Loss<br># 10 1.848200 1.746341<br># 20 1.688300 1.696681<br># 30 1.654500 1.698127<br># 40 1.579400 1.652010<br># 50 1.492600 1.701877</pre><p>We can see the the loss is decreasing. Once the model is fine-tuned to 50 steps, you can test it by applying the LoRA adapters back to the model. You have to load the model as we did in the <a href="#bdbb">beginning</a>.</p><pre>from peft import PeftModel<br><br>peft_model_id = &quot;cp/checkpoint-40&quot;<br>peft_model = PeftModel.from_pretrained(model, peft_model_id, torch_dtype=torch.float16, offload_folder=&quot;lora_results/lora_7/temp&quot;)</pre><p>Lets try to do the inference on the same example that we did it in the beginning and compare the results. Now the model is now summarizing the conversation although it is still not good as we only fine-tuned it for 50 steps.</p><pre>input_prompt = generate_prompt(data_train[50][&quot;dialogue&quot;])<br>input_tokens = tokenizer(input_prompt, return_tensors=&quot;pt&quot;)[&quot;input_ids&quot;].to(&quot;cuda&quot;)<br>with torch.cuda.amp.autocast():<br>    generation_output = peft_model.generate(<br>        input_ids=input_tokens,<br>        max_new_tokens=100,<br>        do_sample=True,<br>        top_k=10,<br>        top_p=0.9,<br>        temperature=0.3,<br>        repetition_penalty=1.15,<br>        num_return_sequences=1,<br>        eos_token_id=tokenizer.eos_token_id,<br>      )<br>op = tokenizer.decode(generation_output[0], skip_special_tokens=True)<br>print(op)<br># Summarize the following:<br>#  Pitt: Hey Teddy! Have you received my message?<br># Teddy: No. An email?<br># Pitt: No. On the FB messenger.<br># Teddy: Let me check.<br># Teddy: Yeah. Ta!<br>#  Summary:   Pitt sent a message to Teddy on Facebook Messenger, but he didn&#39;t receive it yet. <br></pre><p>That was it 😃. LoRA is quite efficient in fine-tuning. We can further speed up the fine-tuning process and even fit the model on smaller GPU using 4-bit quantization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb3f48a557d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps: End-to-end using Kubeflow, MLflow & Seldon: Part 4]]></title>
            <link>https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-4-aa1b1a7a98c3?source=rss-cac8a492636------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa1b1a7a98c3</guid>
            <dc:creator><![CDATA[Rajat Sharma]]></dc:creator>
            <pubDate>Thu, 11 Aug 2022 20:58:22 GMT</pubDate>
            <atom:updated>2022-11-27T05:14:09.311Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-98464b11b946"><em>Part 1</em></a>: Introduction to the pipeline</p><p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-2-6e79dfb54c05"><em>Part 2</em></a>: Set up AKS and Kubeflow on AKS</p><p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-3-6502639ebc6c"><em>Part 3</em></a>: Setup MLflow and Seldon on AKS</p><p><em>Part 4</em>: Training an end-to-end ML pipeline</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*LIYP4bzTopNJz1r1.jpg" /></figure><p>In the last parts, we have set up our infrastructure which we needed to train &amp; deploy our machine learning code. In this part, we’ll train the actual PyTorch model on Kubeflow and logs on the MLflow server. The repo can be found here (<a href="https://github.com/rjtshrm/Cats_Dogs_CNN">https://github.com/rjtshrm/Cats_Dogs_CNN</a>). The whole repo is organised in a modular way such that data preprocessing, training and evaluation are loosely coupled and have different images. This makes to run them on Kubeflow, Azure ML, and SageMaker pipelines.</p><p>Below is the KubeFlow script that triggers the training Pipeline and can be a part of CI/CD or can be triggered manually. I have used persistent volumes to share data between different steps. Kubeflow also provides data sharing options between different steps but since I want to persist data even after the pipeline finishes, I used persistent volumes.</p><pre>import uuid<br>import kfp<br>import kfp.dsl as dsl<br>from kfp.dsl import ContainerOp, component<br>import kubernetes.client as k8s_client<br>import kfp.compiler as compiler<br>from kubernetes.client import V1EnvVar<br><br><br>class MLPipeline:<br><br>    def __init__(self, exp_name, host_url, namespace, pvc_name):<br>        self.exp_name = exp_name<br>        self.client = kfp.Client(host=host_url, namespace=&quot;anonymous&quot;)<br>        print(self.client.list_experiments(namespace=&quot;anonymous&quot;))<br>        self.pvc_name = pvc_name<br><br><br>    def data_generator(self) -&gt; ContainerOp:<br>        op = ContainerOp(<br>            name=&quot;data_generagtor&quot;,<br>            image=&quot;rjtshrm/data_generator:latest&quot;,<br>            arguments=[&quot;--data_dir&quot;, &quot;/tmp/&quot;],<br>            container_kwargs={&#39;env&#39;: [V1EnvVar(&#39;EXP_NAME&#39;, self.exp_name)]}<br>        ).add_volume(k8s_client.V1Volume(<br>            persistent_volume_claim=k8s_client.V1PersistentVolumeClaimVolumeSource(claim_name=self.pvc_name),<br>            name=&#39;kubeflowpvc&#39;))\<br>            .add_volume_mount(k8s_client.V1VolumeMount(mount_path=&#39;/tmp/&#39;, name=&#39;kubeflowpvc&#39;)).set_image_pull_policy(&#39;Always&#39;)<br><br>        return op<br><br>    def trainer(self) -&gt; ContainerOp:<br>        op = ContainerOp(<br>            name=&quot;model_trainer&quot;,<br>            image=&quot;rjtshrm/trainer:latest&quot;,<br>            arguments=[&quot;--mlflow_host&quot;, &quot;http://20.28.195.42/mlflow/&quot;, &quot;--ds_path&quot;, &quot;/tmp/Cat_Dog&quot;, &quot;--epoch&quot;, 1],<br>            container_kwargs={&#39;env&#39;: [V1EnvVar(&#39;EXP_NAME&#39;, self.exp_name)]}<br>        ).add_volume(k8s_client.V1Volume(<br>            persistent_volume_claim=k8s_client.V1PersistentVolumeClaimVolumeSource(claim_name=self.pvc_name),<br>            name=&#39;kubeflowpvc&#39;))\<br>            .add_volume_mount(k8s_client.V1VolumeMount(mount_path=&#39;/tmp/&#39;, name=&#39;kubeflowpvc&#39;)).set_image_pull_policy(&#39;Always&#39;)<br><br>        return op<br><br>    @dsl.pipeline(name=&quot;Classification pipeline&quot;,<br>                  description=&quot;A cat and dog Classification pipeline on Kubefllow&quot;,)<br>    def pipeline(self,):<br>        data_generator_step = self.data_generator()<br>        trainer = self.trainer().after(data_generator_step)<br><br><br>    def pipeline_start(self):<br>        self.client.create_run_from_pipeline_func(<br>            self.pipeline, arguments={}<br>        )<br><br><br>if __name__ == &quot;__main__&quot;:<br>    mlp = MLPipeline(exp_name=str(uuid.uuid4()), host_url=&quot;http://20.28.195.42/pipeline&quot;, namespace=&quot;anonymous&quot;, pvc_name=&quot;kubeflow-pvc&quot;)<br>    mlp.pipeline_start()</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa1b1a7a98c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps: End-to-end using Kubeflow, MLflow & Seldon: Part 3]]></title>
            <link>https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-3-6502639ebc6c?source=rss-cac8a492636------2</link>
            <guid isPermaLink="false">https://medium.com/p/6502639ebc6c</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[seldon]]></category>
            <dc:creator><![CDATA[Rajat Sharma]]></dc:creator>
            <pubDate>Tue, 05 Jul 2022 22:57:03 GMT</pubDate>
            <atom:updated>2022-08-11T20:59:04.858Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-98464b11b946"><em>Part 1</em></a>: Introduction to the pipeline</p><p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-2-6e79dfb54c05"><em>Part 2</em></a>: Set up AKS and Kubeflow on AKS</p><p><em>Part 3</em>: Setup MLflow and Seldon on AKS</p><p><a href="https://medium.com/@rajatsharma_33357/aa1b1a7a98c3"><em>Part 4</em></a>: Training an end-to-end ML pipeline</p><p>In part 3 we are going to set up MLflow and Seldon on the AKS cluster which we have created in <a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-2-6e79dfb54c05"><em>Part 2</em></a><em>. </em>To deploy MLflow on AKS cluster, we first need to create a Docker image for MLflow. I have created a minimalistic Dockerfile to create an image of MLflow.</p><pre>From ubuntu:20.04 </pre><pre>RUN  apt-get update  &amp;&amp; apt install python3-pip libpq-dev -y </pre><pre>WORKDIR /home</pre><pre>COPY . . RUN <br>pip3 install -r requirements.txt </pre><pre>ENTRYPOINT [&quot;mlflow&quot;, &quot;server&quot;]</pre><p>The image based on the above Dockerfile will run an MLflow server provided that you have to pass arguments <strong>--backend-store-uri </strong>(eg.,<strong> </strong>“postgresql://user:pwd@dns_or_ip:5432/db_name?sslmode=require”) to store log metrics, params and <strong>--default-artifact-root </strong>(eg.,<strong> </strong>“wasbs://container@container_ip_or_dns”) to store the model artifacts. For my demo, I have used Psql to store metrics and params, and Azure Containers to store the model atrifacts. The image is also available on the docker hub as <strong>rjtshrm/mlflow:latest</strong>.</p><p>To deploy MLflow on AKS, we first created a new namespace using the below command. Then we create all resources related to MLflow in the same namespace.</p><pre>kubectl create namespace mlflow</pre><p>Now we can deploy the MLflow server using the deployment script. By default, the server will listen on port 5000.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: mlflow-deployment<br>  labels:<br>    app: mlflow<br>spec:<br>  selector:<br>    matchLabels:<br>      app: mlflow<br>  replicas: 1<br>  template:<br>    metadata:<br>      labels:<br>        app: mlflow<br>    spec:<br>      containers:<br>        - name: mlflow<br>          image: rjtshrm/mlflow:latest<br>          args: [&quot;--backend-store-uri&quot;, &quot;postgresql://user:pwd@dns_or_ip:5432/db_name?sslmode=require&quot;, &quot;--default-artifact-root&quot;, &quot;wasbs://container@container_ip_or_dns&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;]<br>          ports:<br>            - containerPort: 5000</pre><p>Once the deployment is created, the server will start on any node on AKS to access it from outside we’ll create a <strong>service</strong> and an <strong>ingress</strong> rule. The service is defined by the following script.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mlflow<br>  labels:<br>    app: mlflow<br>spec:<br>  selector:<br>    app: mlflow<br>  type: ClusterIP<br>  ports:<br>    - port: 5000<br>      targetPort: 5000</pre><p>To create an ingress rule, we first create an ingress controller which we will install using helm (more details <a href="https://docs.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-powershell">here</a>).</p><pre>Namespace=&#39;ingress-basic&#39;</pre><pre>helm repo add ingress-nginx <a href="https://kubernetes.github.io/ingress-nginx">https://kubernetes.github.io/ingress-nginx</a><br>helm repo update</pre><pre>helm install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace $Namespace --set controller.service.annotations.&quot;service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path&quot;=/healthz</pre><p>Once the ingress controller is installed you can verify it using <strong>kubectl get svc ingress -n ingress-basic</strong>. This will display a LoadBalancer with public IP that we can access from anywhere. By default only ports 80, and 443 are exposed by Nginx ingress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5YU0h4M4Iexra2DrkAAg4Q.png" /></figure><p>Now we can create an ingress rule for the service we created for MLflow. The ingress is defined as</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: mlflow-server-ingress<br>  annotations:<br>    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;<br>    nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;<br>    nginx.ingress.kubernetes.io/rewrite-target: /$2<br>spec:<br>  ingressClassName: nginx<br>  rules:<br>    - http:<br>        paths:<br>          - path: /mlflow(/|$)(.*)<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: mlflow<br>                port:<br>                  number: 5000</pre><p>Once the above script is applied, we can access our MLflow server from anywhere. In our demo, we will use it from the Training pipeline to log the metrics, params and model artifacts. You can access the MLflow server from <a href="http://20.213.125.49/mlflow/">http://&lt;public_ip&gt;:80/mlflow/</a> (in my case it is <a href="http://20.213.125.49/mlflow/">http://20.213.125.49/mlflow/</a>, as from the above image 20.213.125.49 is my public IP). The whole code for MLflow can be found <a href="https://github.com/rjtshrm/mlflow-k8">https://github.com/rjtshrm/mlflow-k8</a>.</p><p>Now, to setup up Seldon you can refer <a href="https://www.kubeflow.org/docs/external-add-ons/serving/seldon/">here</a> for a detailed overview. First, create a namespace Seldon and then label the namespace for the inference</p><pre>kubectl create namespace seldon<br>kubectl label namespace seldon serving.kubeflow.org/inferenceservice=enabled</pre><p>Once it is done you can test a Seldon model itself by running the following command in the terminal. Wait until the Seldon deployment status is Available we can test our deployment</p><pre>cat &lt;&lt;EOF | kubectl create -n seldon -f -<br>apiVersion: machinelearning.seldon.io/v1<br>kind: SeldonDeployment<br>metadata:<br>  name: seldon-model<br>spec:<br>  name: test-deployment<br>  predictors:<br>  - componentSpecs:<br>    - spec:<br>        containers:<br>        - image: seldonio/mock_classifier_rest:1.3<br>          name: classifier<br>    graph:<br>      children: []<br>      endpoint:<br>        type: REST<br>      name: classifier<br>      type: MODEL<br>    name: example<br>    replicas: 1<br>EOF</pre><p>To test the deployment, we have to port forward our istio ingress gateway as it is the default gate installed when Kubeflow is installed and since Seldon is part of Kubeflow serving we can route our request from this gateway only for now. To port forward run the following</p><pre>kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath=&#39;{.items[0].metadata.name}&#39;) -n istio-system 8004:80</pre><p>And, to test it run the following command. You should expect a result <em>{“data”:{“names”:[“proba”],”ndarray”:[[0.43782349911420193]]},”meta”:{}}</em></p><pre>curl -s -d &#39;{&quot;data&quot;: {&quot;ndarray&quot;:[[1.0, 2.0, 5.0]]}}&#39;    -X POST http://localhost:8004/seldon/seldon/seldon-model/api/v1.0/predictions    -H &quot;Content-Type: application/json&quot;</pre><p>Now some networking stuff ;). So far we have used Nginx ingress (which is a public ingress) to route our MLflow request to the MLflow server and port forward to get our Kubeflow and Seldon requests because the istio gateway is private. But to access the istio gateway, we can create a new rule in our Nginx ingress to forward all requests to the istio gateway if the URL path wants to access the Kubeflow resources (such as UI, Seldon serving etc.,). We create the following resource in the <strong>istio-system</strong> namespace.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: mlflow-server-ingress<br>  annotations:<br>    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;<br>    nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;<br>spec:<br>  ingressClassName: nginx<br>  rules:<br>    - http:<br>        paths:<br>          - path: /*<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: &quot;istio-ingressgateway&quot;<br>                port:<br>                  number: 80</pre><p>Once you apply this resource you can access the Kubeflow pipeline UI and Seldon from the ingress public IP. If you see the ingress rules just above and one for the above MLflow part, the ingress differentiates between the incoming requests based on the following rules</p><pre>&lt;public_ip&gt;:80/&lt;any_suffixes_after_it&gt;  ------&gt; routes to kubeflow istio gateway</pre><pre>&lt;public_ip&gt;:80/mlflow/&lt;any_suffixes_after_it&gt;  ------&gt; routes to mlflow server</pre><p>We have set up everything for our machine learning development (the infrastructure, the storage and the networking). Now, we are ready with the pipeline development which is done in the next part.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6502639ebc6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps: End-to-end using Kubeflow, Mlflow & Seldon: Part 2]]></title>
            <link>https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-2-6e79dfb54c05?source=rss-cac8a492636------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e79dfb54c05</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <dc:creator><![CDATA[Rajat Sharma]]></dc:creator>
            <pubDate>Tue, 05 Jul 2022 18:46:23 GMT</pubDate>
            <atom:updated>2022-08-11T20:59:14.692Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-98464b11b946"><em>Part 1</em></a>: Introduction to the pipeline</p><p><em>Part 2</em>: Set up AKS and Kubeflow on AKS</p><p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-3-6502639ebc6c"><em>Part 3</em></a>: Setup MlFlow and Seldon on AKS</p><p><a href="https://medium.com/@rajatsharma_33357/aa1b1a7a98c3"><em>Part 4</em></a>: Training an end-to-end ML pipeline</p><p>To set up a Kubeflow on azure we first need to set up a Kubernetes cluster. For that, we will use the Azure Kubernetes Service (AKS). I have used Terraform scripts to set up the AKS. These scripts can be found <a href="https://github.com/rjtshrm/terraform-infra/tree/master/k8/az">https://github.com/rjtshrm/terraform-infra/tree/master/k8/az</a>. Following is the script snippet to create an AKS. We are using the <strong>azurerm_resource_group </strong>resource provided by hashicorp to create the cluster.</p><pre>provider &quot;azurerm&quot; {<br>  features {}<br>}<br><br>resource &quot;azurerm_resource_group&quot; &quot;rg&quot; {<br>  name = &quot;mlops-az&quot;<br>  location = &quot;australiaeast&quot;<br>}<br><br><br>locals {<br>  clustername = &quot;mlops-az-aks&quot;<br>}<br><br>resource &quot;azurerm_kubernetes_cluster&quot; &quot;aks&quot; {<br>  name                = local.clustername<br>  location            = azurerm_resource_group.rg.location<br>  resource_group_name = azurerm_resource_group.rg.name<br>  dns_prefix          = local.clustername<br>  kubernetes_version  = &quot;1.21.7&quot;<br><br>  default_node_pool {<br>    name       = &quot;default&quot;<br>    node_count = 2<br>    vm_size    = &quot;Standard_D4s_v3&quot;<br>    node_labels = {<br>        type: &quot;default&quot;<br>    }<br>  }<br>  <br>  identity {<br>    type = &quot;SystemAssigned&quot;<br>  }<br><br>  tags = {<br>    Environment = &quot;Testing&quot;<br>  }<br>}</pre><p>To create the cluster run <strong>terraform init</strong> and then <strong>terraform apply</strong>. It will take around 10 minutes to set up everything. Once this is done we can proceed with the installation of Kubeflow on the AKS as we need to access the cluster using <strong>kubectl</strong> API. To grant access to the cluster run the following command with the resource and AKS cluster name.</p><pre>az aks get-credentials --resource-group mlops-az --name mlops-az-aks</pre><p>Once this is done, download the <strong>kfctl</strong> from the <a href="https://github.com/kubeflow/kfctl/releases/tag/v1.2.0">Kubeflow releases page</a> and run the following script to deploy Kubeflow on the AKS cluster.</p><pre># The following command is optional. It adds the kfctl binary to your path.<br># If you don&#39;t add kfctl to your path, you must use the full path# each time you run kfctl.<br># Use only alphanumeric characters or - in the directory name.</pre><pre>export PATH=$PATH:&quot;&lt;path-to-kfctl&gt;&quot;</pre><pre># Set KF_NAME to the name of your Kubeflow deployment. You also use this<br># value as directory name when creating your configuration directory.<br># For example, your deployment name can be &#39;my-kubeflow&#39; or &#39;kf-test&#39;.</pre><pre>export KF_NAME=&lt;your choice of name for the Kubeflow deployment&gt;</pre><pre># Set the path to the base directory where you want to store one or more</pre><pre># Kubeflow deployments. For example, /opt/.<br># Then set the Kubeflow application directory for this deployment.</pre><pre>export BASE_DIR=&lt;path to a base directory&gt;<br>export KF_DIR=${BASE_DIR}/${KF_NAME}</pre><pre># Set the configuration file to use when deploying Kubeflow.<br># The following configuration installs Istio by default. Comment out # the Istio components in the config file to skip Istio installation. # See <a href="https://github.com/kubeflow/kubeflow/pull/3663">https://github.com/kubeflow/kubeflow/pull/3663</a></pre><pre>export CONFIG_URI=&quot;https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml&quot; </pre><pre>mkdir -p ${KF_DIR}<br>cd ${KF_DIR}<br>kfctl apply -V -f ${CONFIG_URI}</pre><p>For a more in-depth setup, you can refer to official Kubeflow docs <a href="https://www.kubeflow.org/docs/distributions/azure/deploy/install-kubeflow/">here</a>. Once the script is finished you can run the below command to get the status of Kubeflow resources. Although it may take sometime to get all the resources to start running</p><pre>kubectl get all -n kubeflow</pre><p>To view the Kubeflow UI on the browser, you can forward the port of <strong>svc/istio-ingressgateway </strong>service to 8080 which you can access using localhost or use an ingress at the front of the AKS cluster. We’ll use port forward for now and do the ingress in the next section. You can access the Kubeflow UI in the browser on 127.0.0.1:8080.</p><pre>kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80</pre><p>But if you navigate to the pipeline tab, if you are seeing the error this means that the istio ingree pods is not getting to Kubeflow pods</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*DfdrtZfohoA7WumN_gk-Kw.png" /></figure><p>The workaround is to run the following commands and change <strong>ISTIO_MUTUAL </strong>to<strong> DISABLE </strong>in <strong>spec.trafficPolicy.tls.mode </strong>section (for more details refer <a href="https://github.com/kubeflow/kubeflow/issues/5271">here</a>).</p><pre>kubectl edit destinationrule -n kubeflow ml-pipeline<br>kubectl edit destinationrule -n kubeflow ml-pipeline-ui</pre><p>Now you can deploy the ML pipelines on Kubeflow on AKS cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e79dfb54c05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps: End-to-end using Kubeflow, Mlflow & Seldon: Part 1]]></title>
            <link>https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-98464b11b946?source=rss-cac8a492636------2</link>
            <guid isPermaLink="false">https://medium.com/p/98464b11b946</guid>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[seldon]]></category>
            <dc:creator><![CDATA[Rajat Sharma]]></dc:creator>
            <pubDate>Sun, 03 Jul 2022 21:21:57 GMT</pubDate>
            <atom:updated>2022-08-11T20:59:22.725Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 1</em>: Introduction to the pipeline</p><p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-2-6e79dfb54c05"><em>Part 2</em></a>: Set up AKS and Kubeflow on AKS</p><p><a href="https://medium.com/@rajatsharma_33357/mlops-end-to-end-using-kubeflow-mlflow-seldon-part-3-6502639ebc6c"><em>Part 3</em></a>: Setup MlFlow and Seldon on AKS</p><p><a href="https://medium.com/@rajatsharma_33357/aa1b1a7a98c3"><em>Part 4</em></a>: Training an end-to-end ML pipeline</p><p>MLOps is a very trending topic among Machine Learning engineers and data scientists. It is basically setting up a workflow for training, testing and making the model available to production.</p><p><strong>Kubeflow</strong>: It is a machine learning platform to create pipelines on the Kubernetes cluster in a scalable manner. We can orchestrate full ML workflows that are independent of each other and run them as a graph such as the output of one step being used as input to other. One of the advantages of Kubeflow is that it takes full advantage of Kubernetes such as Persistent Volumes, Node Selections etc. In our project as well we are running mlflow on one node, some tasks (that do not require GPU) on a normal node group and training on gpu-based machines. In this way, we can utilize the node&#39;s maximum and will also help in minimising the costs.</p><p><strong>Mlflow: </strong>Mlflow is a tool to track the experiments, reproducibility of the results, package the models and model versioning. It has support for all the major ML frameworks such as Sklearn, Pytorch, Tensorflow etc. In our use case, we are using the Mlflow to track the experiments based on different parameters and corresponding output metrics. We also save the model artifacts to Mlflow model registry so that it can be later used for the inference. For more details refer <a href="https://mlflow.org/docs/latest/index.html">https://mlflow.org/docs/latest/index.html</a>.</p><p>So why are we using Mlflow? We could have also used Kubeflow metric but they are updated once the component run is finished but with the Mlflow we can see realtime updates and many more. We run our Mlflow server on one of the nodes on Kubeflow and is using Azure PSQL and Azure Containers to save logs and the model artifacts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*JPQwJHqTup7I1vv-m5Qj0Q.png" /><figcaption>End-to-End MlOps</figcaption></figure><p><strong>Seldon: </strong>Seldon provides the way to put the model or a wrapper around it as an endpoint for production use. It runs on the Kubernetes cluster. The important aspect of using Seldon is that it is a production-ready tool. We can scale the model server as per the demand.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98464b11b946" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>