<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sathiya Keerthi on Medium]]></title>
        <description><![CDATA[Stories by Sathiya Keerthi on Medium]]></description>
        <link>https://medium.com/@sathiyakeerthi?source=rss-90dfd0aea7b1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*61x33CabwjCBhBMn.jpg</url>
            <title>Stories by Sathiya Keerthi on Medium</title>
            <link>https://medium.com/@sathiyakeerthi?source=rss-90dfd0aea7b1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:18:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sathiyakeerthi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part 4: Building an LLMOps Pipeline: Fine-Tuning and Serving Your LLM]]></title>
            <link>https://medium.com/@sathiyakeerthi/part-4-building-an-llmops-pipeline-fine-tuning-and-serving-your-llm-2fb5e60d8ecb?source=rss-90dfd0aea7b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fb5e60d8ecb</guid>
            <dc:creator><![CDATA[Sathiya Keerthi]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 14:32:47 GMT</pubDate>
            <atom:updated>2025-04-14T14:32:47.247Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Bring your LLM to life with fine-tuning and deployment on GKE.</em></h4><p>Welcome to the grand finale of our four-part series on building an LLMOps pipeline with <strong>Apache Airflow</strong> and <strong>Google Kubernetes Engine (GKE)</strong>! We’ve set up Google Cloud (), spun up a GPU-powered GKE cluster (), and orchestrated data tasks with Airflow (). Now, it’s time to tie it all together: fine-tuning our large language model (LLM) and serving it for real-world use.</p><p>In Part 3, we built containers for downloading data, preparing a dataset, and fine-tuning <strong>Gemma-2–9b-it</strong> for sentiment analysis on movie reviews. Here, we’ll run those tasks, deploy the fine-tuned model for inference, and orchestrate everything with an Airflow DAG. By the end, you’ll have a fully operational pipeline that takes raw data to a deployed LLM, ready to predict sentiments like “POSITIVE” or “NEGATIVE” for movie reviews.</p><p>This guide is for LLM engineers, data scientists, or anyone eager to see an LLMOps pipeline in action. All code is in , and the lays out the journey. Missed a step? Catch up with , , or .</p><p><em>What’s the most exciting part of deploying an LLM for you? Share in the comments!</em></p><h3>From Data to Model: The Fine-Tuning Task</h3><p>Our pipeline’s heart is fine-tuning Gemma-2–9b-it to excel at sentiment analysis. In Part 3, we created a container for this task, powered by the finetuning.py script. It uses <strong>Low-Rank Adaptation (LoRA)</strong> to efficiently tweak the model’s weights, training on a dataset of 1,000 movie reviews formatted as question-context-answer triplets (e.g., “Question: Review analysis…\nContext: Great film!\nAnswer: POSITIVE”).</p><p>Let’s run this task on GKE’s training node pool, equipped with an Nvidia L4 GPU. First, we build and push the container to Artifact Registry, then execute it as a Kubernetes job. Here’s how:</p><h4>Step 1: Build the Fine-Tuning Container</h4><p>The fine-tuning task’s Dockerfile, from Part 3, uses an Nvidia CUDA base image and installs dependencies like transformers and peft. Navigate to the task directory:</p><pre>cd llmops-pipeline/part3-airflow-tasks/fine-tuning</pre><p>Build and push the container to Artifact Registry (replace &lt;PROJECT_ID&gt; and &lt;LOCATION&gt; with your Google Cloud details):</p><pre>gcloud builds submit --tag &lt;LOCATION&gt;-docker.pkg.dev/&lt;PROJECT_ID&gt;/llmops-airflow/fine-tuning:latest .</pre><p>This creates an image tagged like <strong><em>us-central1-docker.pkg.dev/my-project/llmops-airflow/fine-tuning:latest.</em></strong></p><h4>Step 2: Run the Fine-Tuning Job</h4><p>We’ll deploy the fine-tuning task as a Kubernetes job on GKE’s training node pool. The job spec, from Part 3, requests one L4 GPU and mounts our service account for GCS access. Apply it with:</p><pre>kubectl apply -f llmops-pipeline/part3-airflow-tasks/fine-tuning/fine-tuning-job.yaml</pre><p>Here’s a peek at the job spec (<strong><em>fine-tuning-job.yaml</em></strong>):</p><pre>apiVersion: batch/v1<br>kind: Job<br>metadata:<br>  name: fine-tuning-job<br>spec:<br>  template:<br>    spec:<br>      nodeSelector:<br>        cloud.google.com/gke-accelerator: nvidia-tesla-l4<br>      containers:<br>      - name: fine-tuning<br>        image: &lt;LOCATION&gt;-docker.pkg.dev/&lt;PROJECT_ID&gt;/llmops-airflow/fine-tuning:latest<br>        env:<br>        - name: BUCKET_DATA_NAME<br>          value: &quot;llmops-airflow-data-&lt;SUFFIX&gt;&quot;<br>        - name: MODEL_PATH<br>          value: &quot;./output&quot;<br>        resources:<br>          limits:<br>            nvidia.com/gpu: 1<br>      restartPolicy: Never</pre><p>This job runs <strong>finetuning.py</strong>, pulling the dataset from GCS (<strong>gs://llmops-airflow-data-&lt;SUFFIX&gt;/prepared_data.jsonl</strong>), fine-tuning Gemma for one epoch, and saving the model to <strong>./output</strong> before uploading to GCS (<strong>gs://llmops-airflow-data-&lt;SUFFIX&gt;/fine_tuned_model</strong>).</p><pre>kubectl get jobs fine-tuning-job</pre><p>Once completed, verify the model in GCS:</p><pre>gsutil ls gs://llmops-airflow-data-&lt;SUFFIX&gt;/fine_tuned_model</pre><p>The fine-tuned model is now ready — trained to predict movie sentiments with precision!</p><h3>Serving the Model: Inference with vLLM</h3><p>Fine-tuning is only half the story. To make our model useful, we need to serve it for inference, letting it answer queries like “Is this review positive?” We’ll use <strong>vLLM</strong>, a high-performance library for LLM inference, to deploy the model on GKE’s inference node pool (also with L4 GPUs).</p><h4>Step 3: Build the Inference Container</h4><p>Let’s create a container to serve the fine-tuned model using vLLM. Save this Dockerfile in <strong>llmops-pipeline/inference/Dockerfile</strong>:</p><pre>FROM python:3.10-slim<br>RUN pip install vllm==0.5.4<br>COPY requirements.txt .<br>RUN pip install -r requirements.txt<br>CMD [&quot;vllm&quot;, &quot;serve&quot;, &quot;/model&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;8000&quot;]</pre><p>The <strong>requirements.txt </strong>includes minimal dependencies:</p><pre>torch==2.3.1<br>transformers==4.42.4</pre><p>Build and push the container:</p><pre>cd llmops-pipeline/inference<br>gcloud builds submit --tag &lt;LOCATION&gt;-docker.pkg.dev/&lt;PROJECT_ID&gt;/llmops-airflow/inference:latest .</pre><h4>Step 4: Deploy the Inference Service</h4><p>We’ll deploy vLLM as a Kubernetes service, mounting the fine-tuned model from GCS. Create a deployment spec (<strong><em>inference-deployment.yaml</em></strong>):</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: inference-deployment<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: inference<br>  template:<br>    metadata:<br>      labels:<br>        app: inference<br>    spec:<br>      nodeSelector:<br>        cloud.google.com/gke-accelerator: nvidia-tesla-l4<br>      containers:<br>      - name: inference<br>        image: &lt;LOCATION&gt;-docker.pkg.dev/&lt;PROJECT_ID&gt;/llmops-airflow/inference:latest<br>        env:<br>        - name: GOOGLE_APPLICATION_CREDENTIALS<br>          value: &quot;/secrets/gcp-credentials.json&quot;<br>        volumeMounts:<br>        - name: model<br>          mountPath: &quot;/model&quot;<br>        - name: gcp-credentials<br>          mountPath: &quot;/secrets&quot;<br>          readOnly: true<br>        resources:<br>          limits:<br>            nvidia.com/gpu: 1<br>      volumes:<br>      - name: model<br>        persistentVolumeClaim:<br>          claimName: model-pvc<br>      - name: gcp-credentials<br>        secret:<br>          secretName: gcp-credentials<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: inference-service<br>spec:<br>  selector:<br>    app: inference<br>  ports:<br>  - protocol: TCP<br>    port: 80<br>    targetPort: 8000<br>  type: LoadBalancer</pre><p>Create a PersistentVolumeClaim to mount the model from GCS (<strong>model-pvc.yaml</strong>):</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: model-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 100Gi</pre><p>Apply these configs:</p><pre>kubectl apply -f llmops-pipeline/inference/model-pvc.yaml<br>kubectl apply -f llmops-pipeline/inference/inference-deployment.yaml</pre><p>Copy the model from GCS to the PVC (you may need a temporary pod to run this):</p><pre>gsutil cp -r gs://llmops-airflow-data-&lt;SUFFIX&gt;/fine_tuned_model/* /path/to/pvc</pre><p>Get the external IP for the service:</p><pre>kubectl get svc inference-service</pre><p>Test the model with a curl request (replace &lt;EXTERNAL_IP&gt;):</p><pre>curl -X POST http://&lt;EXTERNAL_IP&gt;/v1/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;model&quot;: &quot;fine_tuned_model&quot;, &quot;prompt&quot;: &quot;Question: Review analysis for movie\nContext: Great film!\nAnswer:&quot;, &quot;max_tokens&quot;: 10}&#39;</pre><p>Expect output like:</p><pre>{<br>  &quot;choices&quot;: [<br>    {<br>      &quot;text&quot;: &quot;POSITIVE&quot;<br>    }<br>  ]<br>}</pre><p>Your model is live, serving sentiment predictions!</p><h3>Orchestrating with Airflow</h3><p>Now, let’s stitch everything together with an Airflow DAG to automate the pipeline: download data, prepare it, fine-tune the model, and deploy it. Save this as <strong>llmops_pipeline.py</strong> in Airflow’s DAGs folder (<strong>/opt/airflow/dags</strong>):</p><pre>from datetime import datetime<br>from airflow import DAG<br>from airflow.providers.kubernetes.operators.kubernetes import KubernetesPodOperator<br>from airflow.operators.dummy import DummyOperator<br><br>default_args = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;depends_on_past&#39;: False,<br>    &#39;retries&#39;: 1,<br>}<br><br>with DAG(<br>    &#39;llmops_pipeline&#39;,<br>    default_args=default_args,<br>    description=&#39;LLMOps pipeline for fine-tuning and serving&#39;,<br>    schedule_interval=None,<br>    start_date=datetime(2025, 1, 1),<br>    catchup=False,<br>) as dag:<br>    start = DummyOperator(task_id=&#39;start&#39;)<br>    <br>    download_data = KubernetesPodOperator(<br>        task_id=&#39;download_data&#39;,<br>        name=&#39;download-data&#39;,<br>        namespace=&#39;default&#39;,<br>        image=&#39;&lt;LOCATION&gt;-docker.pkg.dev/&lt;PROJECT_ID&gt;/llmops-airflow/download-data:latest&#39;,<br>        env_vars={<br>            &#39;BUCKET_DATA_NAME&#39;: &#39;llmops-airflow-data-&lt;SUFFIX&gt;&#39;,<br>        },<br>        get_logs=True,<br>    )<br>    <br>    prepare_data = KubernetesPodOperator(<br>        task_id=&#39;prepare_data&#39;,<br>        name=&#39;prepare-data&#39;,<br>        namespace=&#39;default&#39;,<br>        image=&#39;&lt;LOCATION&gt;-docker.pkg.dev/&lt;PROJECT_ID&gt;/llmops-airflow/prepare-data:latest&#39;,<br>        env_vars={<br>            &#39;BUCKET_DATA_NAME&#39;: &#39;llmops-airflow-data-&lt;SUFFIX&gt;&#39;,<br>        },<br>        get_logs=True,<br>    )<br>    <br>    fine_tune = KubernetesPodOperator(<br>        task_id=&#39;fine_tune&#39;,<br>        name=&#39;fine-tune&#39;,<br>        namespace=&#39;default&#39;,<br>        image=&#39;&lt;LOCATION&gt;-docker.pkg.dev/&lt;PROJECT_ID&gt;/llmops-airflow/fine-tuning:latest&#39;,<br>        env_vars={<br>            &#39;BUCKET_DATA_NAME&#39;: &#39;llmops-airflow-data-&lt;SUFFIX&gt;&#39;,<br>            &#39;MODEL_PATH&#39;: &#39;./output&#39;,<br>        },<br>        resources={<br>            &#39;limit&#39;: {&#39;nvidia.com/gpu&#39;: &#39;1&#39;},<br>        },<br>        node_selector={&#39;cloud.google.com/gke-accelerator&#39;: &#39;nvidia-tesla-l4&#39;},<br>        get_logs=True,<br>    )<br>    <br>    end = DummyOperator(task_id=&#39;end&#39;)<br>    <br>    start &gt;&gt; download_data &gt;&gt; prepare_data &gt;&gt; fine_tune &gt;&gt; end</pre><p>Upload to Airflow’s GCS bucket:</p><pre>gsutil cp llmops_pipeline.py gs://&lt;COMPOSER_BUCKET&gt;/dags/</pre><p>Log into Airflow’s UI (from Part 3), trigger the llmops_pipeline DAG, and watch it run each task sequentially. Once complete, the fine-tuned model is in GCS, ready for the inference service we deployed.</p><h3>Wrapping Up the LLMOps Journey</h3><p>Congratulations — you’ve built an end-to-end LLMOps pipeline! From raw movie review data to a deployed model serving predictions, you’ve harnessed Google Cloud, GKE, Airflow, and vLLM to make it happen. The pipeline is flexible: tweak finetuning.py’s settings (in the repo) or scale the inference deployment for more users.</p><p>Explore the to customize tasks or try new datasets. Revisit for cloud setup, for GKE, or for Airflow details. The ties it all together.</p><p>This pipeline is just the start. Maybe you’ll fine-tune for a new task (like your flowertune-llm ideas for autism support) or deploy on another cloud. Whatever’s next, you’ve got the tools to build AI that scales.</p><p><em>Enjoyed seeing the pipeline come alive? Clap and share your LLMOps wins below!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fb5e60d8ecb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 3:Building an LLMOps Pipeline: Orchestrating Tasks with Airflow]]></title>
            <link>https://medium.com/@sathiyakeerthi/part-3-building-an-llmops-pipeline-orchestrating-tasks-with-airflow-028970568f32?source=rss-90dfd0aea7b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/028970568f32</guid>
            <dc:creator><![CDATA[Sathiya Keerthi]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 14:28:08 GMT</pubDate>
            <atom:updated>2025-04-14T14:28:08.720Z</atom:updated>
            <content:encoded><![CDATA[<h4>Install Airflow on GKE and build containers to process data and fine-tune an LLM.</h4><p>Welcome to the third stop on our journey to build an LLMOps pipeline with <strong>Apache Airflow</strong> and <strong>Google Kubernetes Engine (GKE)</strong>! In , we set up Google Cloud, and in , we built a GPU-powered GKE cluster with storage and Kubernetes resources. Now, it’s time to introduce the conductor of our pipeline: Airflow.</p><p>This is Part 3 of our four-part series, where we’ll install Airflow on GKE and create the tasks that power our pipeline — downloading movie review data, preparing it, and fine-tuning the Gemma-2–9b-it model for sentiment analysis. These tasks are the heart of our LLMOps workflow, setting the stage for deployment in Part 4.</p><p>New to Airflow? No problem — this guide is for LLM engineers, platform pros, and data scientists curious about automating LLM workflows. All code is in , and the maps out the full journey. Missed earlier parts? Catch up with and .</p><p><em>What’s your favorite tool for orchestrating ML pipelines? Share in the comments!</em></p><h3>Why Airflow for LLMOps?</h3><p>Airflow is a rockstar for pipeline orchestration. Its <strong>Directed Acyclic Graphs (DAGs)</strong> let you define workflows as code, scheduling tasks like data processing and model training with precision. Running Airflow on GKE gives us Kubernetes’ scalability and GKE’s GPU power — perfect for LLM tasks that need heavy compute.</p><p>In this part, we’ll:</p><ul><li>Install Airflow on GKE using Helm.</li><li>Configure Airflow’s UI and connections.</li><li>Build three containerized tasks: data download, data preparation, and fine-tuning.</li></ul><p>By the end, your pipeline will be ready to orchestrate, just waiting for the DAG to tie it together. Let’s dive in!</p><h3>Step 1: Install Airflow on GKE</h3><p>Airflow runs as a Kubernetes application on our GKE cluster. We’ll use <strong>Helm</strong>, Kubernetes’ package manager, to deploy it. First, add the Airflow Helm repository in Cloud Shell:</p><pre>helm repo add apache-airflow https://airflow.apache.org<br>helm repo update</pre><p>Now, install Airflow in the airflow namespace (created in Part 2):</p><pre>helm repo add apache-airflow https://airflow.apache.org<br>helm repo update</pre><p>The <strong>values.yaml</strong> file customizes Airflow for our needs — using KubernetesExecutor to run tasks as pods and mounting GCS buckets for logs and DAGs. Here’s the full configuration:</p><pre>config:<br>  webserver:<br>    expose_config: true<br>webserver:<br>  service:<br>    type: LoadBalancer<br>  podAnnotations:<br>    gke-gcsfuse/volumes: &quot;true&quot;<br>executor: KubernetesExecutor<br>extraEnv: |<br>  - name: AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL<br>    value: &quot;30&quot;<br>logs:<br>  persistence:<br>    enabled: true<br>    existingClaim: &quot;airflow-logs&quot;<br>dags:<br>  persistence:<br>    enabled: true<br>    existingClaim: &quot;airflow-dags&quot;<br>scheduler:<br>  podAnnotations:<br>    gke-gcsfuse/volumes: &quot;true&quot;<br>triggerer:<br>  podAnnotations:<br>    gke-gcsfuse/volumes: &quot;true&quot;<br>workers:<br>  podAnnotations:<br>    gke-gcsfuse/volumes: &quot;true&quot;</pre><p>This sets:</p><ul><li><strong>KubernetesExecutor </strong>to run tasks in pods.</li><li><strong>LoadBalancer</strong> for webserver access.</li><li>Persistent volumes (<strong>airflow-logs, airflow-dags</strong>) linked to GCS buckets.</li><li>Annotations for GCS FUSE to mount buckets.</li></ul><p>Find the full <strong>values.yaml</strong> in the . To access the Airflow UI, get the webserver’s IP:</p><pre>kubectl -n airflow get svc/airflow-webserver --output jsonpath=&#39;{.status.loadBalancer.ingress[0].ip}&#39;</pre><p>Open <strong>http://&lt;IP&gt;:8080 </strong>in your browser, log in with <strong>admin/admin</strong>, and you’ll see Airflow’s dashboard. It’s where we’ll manage our pipeline in Part 4.</p><p><em>Ever customized Airflow? What’s your favorite tweak?</em></p><h3>Step 2: Configure Airflow</h3><p>Before building tasks, configure Airflow to connect to Google Cloud and store variables. In the Airflow UI:</p><p>Go to <strong>Admin</strong> → <strong>Connections</strong>, add a new connection:</p><ul><li><strong>Conn Id</strong>: <strong>google_cloud_default</strong></li><li><strong>Conn Type</strong>: Google Cloud</li><li>Leave other fields default (it uses your GKE service account).</li></ul><p>Go to <strong>Admin</strong> → <strong>Variables</strong>, add:</p><ul><li><strong>BUCKET_DATA_NAME</strong>: Your data bucket (e.g., <strong>llmops-airflow-data-&lt;suffix&gt;</strong>).</li><li><strong>GCP_PROJECT_ID</strong>: Your project ID (e.g., <strong>llmops-pipeline-123456</strong>).</li><li><strong>HF_TOKEN:</strong> Your Hugging Face read token.</li><li><strong>KAGGLE_USERNAME</strong>: From <strong>kaggle.json</strong>.</li><li><strong>KAGGLE_KEY:</strong> From <strong>kaggle.json</strong>.</li></ul><p>These variables let tasks access GCS, Hugging Face, and Kaggle securely. You can also set them via Cloud Shell for automation — check the repo for a script.</p><h3>Step 3: Build Pipeline Tasks</h3><p>Our pipeline has three tasks, each running in a container orchestrated by Airflow. We’ll build them using Python scripts, Dockerfiles, and Google Cloud Build, storing images in Artifact Registry. Let’s create each one.</p><h4>Task 1: Data Download</h4><p>This task fetches the Rotten Tomatoes movie review dataset from Kaggle and uploads it to GCS. Create the files:</p><pre>kubectl -n airflow get svc/airflow-webserver --output jsonpath=&#39;{.status.loadBalancer.ingress[0].ip}&#39;</pre><p><strong>dataset-download.py</strong>:</p><pre>import os<br>import kagglehub<br>from google.cloud import storage<br><br>KAGGLE_USERNAME = os.getenv(&quot;KAGGLE_USERNAME&quot;)<br>KAGGLE_KEY = os.getenv(&quot;KAGGLE_KEY&quot;)<br>BUCKET_DATA_NAME = os.getenv(&quot;BUCKET_DATA_NAME&quot;)<br><br>def upload_blob(bucket_name, source_file_name, destination_blob_name):<br>    storage_client = storage.Client()<br>    bucket = storage_client.bucket(bucket_name)<br>    blob = bucket.blob(destination_blob_name)<br>    blob.upload_from_filename(source_file_name)<br>    print(f&quot;File {source_file_name} uploaded to {destination_blob_name}.&quot;)<br><br>path = kagglehub.dataset_download(&quot;priyamchoksi/rotten-tomato-movie-reviews-1-44m-rows&quot;)<br>print(&quot;Path to dataset files:&quot;, path)<br>destination_blob_name = &quot;rotten_tomatoes_movie_reviews.csv&quot;<br>source_file_name = f&quot;{path}/{destination_blob_name}&quot;<br>upload_blob(BUCKET_DATA_NAME, source_file_name, destination_blob_name)</pre><p><strong>Dockerfile</strong>:</p><pre>FROM python:3.13.0-slim-bookworm<br>WORKDIR /app<br>COPY requirements.txt .<br>RUN pip install --no-cache-dir --upgrade pip &amp;&amp; pip install --no-cache-dir -r requirements.txt<br>COPY dataset-download.py .<br>CMD [&quot;python&quot;, &quot;dataset-download.py&quot;]</pre><p><strong>requirements.txt</strong>:</p><pre>google-cloud-storage==2.19.0<br>kagglehub==0.3.4</pre><p>Build and push to Artifact Registry:</p><pre>gcloud builds submit --tag ${REGION}-docker.pkg.dev/${DEVSHELL_PROJECT_ID}/${REPO_NAME}/data</pre><p>This creates a container that downloads the dataset and stores it in <strong>gs://&lt;BUCKET_DATA_NAME&gt;/rotten_tomatoes_movie_reviews.csv.</strong></p><h4>Task 2: Data Preparation</h4><p>This task transforms the dataset into a question-answering format, limiting to 1,000 rows for efficiency. Create the files:</p><pre>cd .. ; mkdir 2-data-preparation<br>cd 2-data-preparation</pre><p><strong>data-preparation.py</strong>:</p><pre>import os<br>import pandas as pd<br>import gcsfs<br>import json<br>from datasets import Dataset<br><br>GCP_PROJECT_ID = os.getenv(&quot;GCP_PROJECT_ID&quot;)<br>BUCKET_DATA_NAME = os.getenv(&quot;BUCKET_DATA_NAME&quot;)<br>DATASET_NAME = os.getenv(&quot;DATASET_NAME&quot;, &quot;rotten_tomatoes_movie_reviews.csv&quot;)<br>PREPARED_DATASET_NAME = os.getenv(&quot;PREPARED_DATA_URL&quot;, &quot;prepared_data.jsonl&quot;)<br>DATASET_LIMIT = int(os.getenv(&quot;DATASET_LIMIT&quot;, &quot;100&quot;))<br>DATASET_URL = f&quot;gs://{BUCKET_DATA_NAME}/{DATASET_NAME}&quot;<br>PREPARED_DATASET_URL = f&quot;gs://{BUCKET_DATA_NAME}/{PREPARED_DATASET_NAME}&quot;<br><br>print(f&quot;Loading dataset from {DATASET_URL}...&quot;)<br><br>def transform(data):<br>    question = f&quot;Review analysis for movie &#39;{data[&#39;id&#39;]}&#39;&quot;<br>    context = data[&#39;reviewText&#39;]<br>    answer = data[&#39;scoreSentiment&#39;]<br>    template = &quot;Question: {question}\nContext: {context}\nAnswer: {answer}&quot;<br>    return {&#39;text&#39;: template.format(question=question, context=context, answer=answer)}<br><br>try:<br>    df = pd.read_csv(DATASET_URL, nrows=DATASET_LIMIT)<br>    print(f&quot;Dataset loaded successfully.&quot;)<br>    df = df.dropna(subset=[&#39;id&#39;, &#39;reviewText&#39;, &#39;scoreSentiment&#39;])<br>    transformed_data = df.apply(transform, axis=1).tolist()<br>    transformed_df = pd.DataFrame(transformed_data)<br>    dataset = Dataset.from_pandas(transformed_df)<br>    with gcsfs.GCSFileSystem(project=GCP_PROJECT_ID).open(PREPARED_DATASET_URL, &#39;w&#39;) as f:<br>        for item in dataset:<br>            f.write(json.dumps(item) + &quot;\n&quot;)<br>    print(f&quot;Prepared dataset saved to {PREPARED_DATASET_URL}&quot;)<br>except Exception as e:<br>    print(f&quot;Error during data loading or preprocessing: {e}&quot;)<br>    import traceback<br>    print(traceback.format_exc())</pre><p><strong>Dockerfile</strong>:</p><pre>FROM python:3.13.0-slim-bookworm<br>WORKDIR /app<br>COPY requirements.txt .<br>RUN pip install --no-cache-dir --upgrade pip &amp;&amp; pip install --no-cache-dir -r requirements.txt<br>COPY data-preparation.py .<br>CMD [&quot;python&quot;, &quot;data-preparation.py&quot;]</pre><p><strong>requirements.txt</strong>:</p><pre>datasets==3.1.0<br>gcsfs==2024.9.0<br>pandas==2.2.3</pre><p>Build and push:</p><pre>gcloud builds submit --tag ${REGION}-docker.pkg.dev/${DEVSHELL_PROJECT_ID}/${REPO_NAME}/data-prep</pre><p>This outputs gs://&lt;BUCKET_DATA_NAME&gt;/prepared_data.jsonl, formatted for fine-tuning.</p><h4>Task 3: Model Fine-Tuning</h4><p>This task fine-tunes Gemma-2–9b-it using Low-Rank Adaptation (LoRA) for efficiency. Create the files:</p><pre>cd .. ; mkdir 3-fine-tuning<br>cd 3-fine-tuning</pre><p><strong>finetuning.py</strong>:</p><pre>import os<br>import torch<br>import bitsandbytes<br>from accelerate import Accelerator<br>from datasets import Dataset, load_dataset<br>from peft import LoraConfig, PeftModel<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from trl import DataCollatorForCompletionOnlyLM, SFTConfig, SFTTrainer<br>from google.cloud import storage<br><br>BUCKET_DATA_NAME = os.environ[&quot;BUCKET_DATA_NAME&quot;]<br>PREPARED_DATA_URL = os.getenv(&quot;PREPARED_DATA_URL&quot;, &quot;prepared_data.jsonl&quot;)<br>new_model = os.getenv(&quot;NEW_MODEL_NAME&quot;, &quot;fine_tuned_model&quot;)<br>model_name = os.getenv(&quot;MODEL_ID&quot;, &quot;google/gemma-2-9b-it&quot;)<br>save_model_path = os.getenv(&quot;MODEL_PATH&quot;, &quot;./output&quot;)<br><br>print(&quot;Loading tokenizer...&quot;)<br>tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)<br>tokenizer.pad_token = tokenizer.eos_token<br>tokenizer.padding_side = &quot;right&quot;<br>print(&quot;Tokenizer loaded successfully!&quot;)<br><br>EOS_TOKEN = tokenizer.eos_token<br>dataset = load_dataset(<br>    &quot;json&quot;, data_files=f&quot;gs://{BUCKET_DATA_NAME}/{PREPARED_DATA_URL}&quot;, split=&quot;train&quot;)<br>print(dataset)<br><br>lora_r = int(os.getenv(&quot;LORA_R&quot;, &quot;8&quot;))<br>lora_alpha = int(os.getenv(&quot;LORA_ALPHA&quot;, &quot;16&quot;))<br>lora_dropout = float(os.getenv(&quot;LORA_DROPOUT&quot;, &quot;0.1&quot;))<br>num_train_epochs = int(os.getenv(&quot;EPOCHS&quot;, 1))<br>fp16 = False<br>bf16 = False<br>per_device_train_batch_size = int(os.getenv(&quot;TRAIN_BATCH_SIZE&quot;, &quot;1&quot;))<br>per_device_eval_batch_size = 1<br>gradient_accumulation_steps = int(os.getenv(&quot;GRADIENT_ACCUMULATION_STEPS&quot;, &quot;1&quot;))<br>gradient_checkpointing = True<br>max_grad_norm = 0.3<br>learning_rate = 2e-4<br>weight_decay = 0.001<br>optim = &quot;paged_adamw_32bit&quot;<br>lr_scheduler_type = &quot;cosine&quot;<br>max_steps = -1<br>warmup_ratio = 0.03<br>group_by_length = True<br>save_strategy = os.getenv(&quot;CHECKPOINT_SAVE_STRATEGY&quot;, &quot;steps&quot;)<br>save_total_limit = int(os.getenv(&quot;CHECKPOINT_SAVE_TOTAL_LIMIT&quot;, &quot;5&quot;))<br>save_steps = int(os.getenv(&quot;CHECKPOINT_SAVE_STEPS&quot;, &quot;1000&quot;))<br>logging_steps = 50<br>max_seq_length = int(os.getenv(&quot;MAX_SEQ_LENGTH&quot;, &quot;512&quot;))<br>packing = False<br><br>print(f&quot;Loading base model started&quot;)<br>model = AutoModelForCausalLM.from_pretrained(<br>    attn_implementation=&quot;eager&quot;,<br>    pretrained_model_name_or_path=model_name,<br>    torch_dtype=torch.float16,<br>)<br>model.config.use_cache = False<br>model.config.pretraining_tp = 1<br>print(&quot;Loading base model completed&quot;)<br><br>print(f&quot;Configuring fine tuning started&quot;)<br>peft_config = LoraConfig(<br>    lora_alpha=lora_alpha,<br>    lora_dropout=lora_dropout,<br>    r=lora_r,<br>    bias=&quot;none&quot;,<br>    task_type=&quot;CAUSAL_LM&quot;,<br>    target_modules=[<br>        &quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;, &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;,<br>    ],<br>)<br><br>training_arguments = SFTConfig(<br>    bf16=bf16,<br>    dataset_kwargs={&quot;add_special_tokens&quot;: False, &quot;append_concat_token&quot;: False},<br>    dataset_text_field=&quot;text&quot;,<br>    disable_tqdm=True,<br>    fp16=fp16,<br>    gradient_accumulation_steps=gradient_accumulation_steps,<br>    gradient_checkpointing=gradient_checkpointing,<br>    gradient_checkpointing_kwargs={&quot;use_reentrant&quot;: False},<br>    group_by_length=group_by_length,<br>    log_on_each_node=False,<br>    logging_steps=logging_steps,<br>    learning_rate=learning_rate,<br>    lr_scheduler_type=lr_scheduler_type,<br>    max_grad_norm=max_grad_norm,<br>    max_seq_length=max_seq_length,<br>    max_steps=max_steps,<br>    num_train_epochs=num_train_epochs,<br>    optim=optim,<br>    output_dir=save_model_path,<br>    packing=packing,<br>    per_device_train_batch_size=per_device_train_batch_size,<br>    save_strategy=save_strategy,<br>    save_steps=save_steps,<br>    save_total_limit=save_total_limit,<br>    warmup_ratio=warmup_ratio,<br>    weight_decay=weight_decay,<br>)<br><br>print(f&quot;Creating trainer started&quot;)<br>trainer = SFTTrainer(<br>    model=model,<br>    train_dataset=dataset,<br>    peft_config=peft_config,<br>    dataset_text_field=&quot;text&quot;,<br>    max_seq_length=max_seq_length,<br>    tokenizer=tokenizer,<br>    args=training_arguments,<br>    packing=packing,<br>)<br>print(f&quot;Creating trainer completed&quot;)<br><br>print(&quot;Starting fine-tuning...&quot;)<br>trainer.train()<br>print(&quot;Fine-tuning completed.&quot;)<br><br>print(&quot;Saving new model started&quot;)<br>trainer.model.save_pretrained(new_model)<br>print(&quot;Saving new model completed&quot;)<br><br>print(f&quot;Merging the new model with base model started&quot;)<br>base_model = AutoModelForCausalLM.from_pretrained(<br>    low_cpu_mem_usage=True,<br>    pretrained_model_name_or_path=model_name,<br>    return_dict=True,<br>    torch_dtype=torch.float16,<br>)<br>model = PeftModel.from_pretrained(model=base_model, model_id=new_model)<br>model = model.merge_and_unload()<br>print(f&quot;Merging the new model with base model completed&quot;)<br><br>accelerator = Accelerator()<br>print(f&quot;Accelerate unwrap model started&quot;)<br>unwrapped_model = accelerator.unwrap_model(model)<br>print(f&quot;Accelerate unwrap model completed&quot;)<br>print(f&quot;Save unwrapped model started&quot;)<br>unwrapped_model.save_pretrained(<br>    is_main_process=accelerator.is_main_process,<br>    save_directory=save_model_path,<br>    save_function=accelerator.save,<br>)<br>print(f&quot;Save unwrapped model completed&quot;)<br>print(f&quot;Save new tokenizer started&quot;)<br>if accelerator.is_main_process:<br>    tokenizer.save_pretrained(save_model_path)<br>print(f&quot;Save new tokenizer completed&quot;)<br><br>def upload_to_gcs(bucket_name, model_dir):<br>    storage_client = storage.Client()<br>    bucket = storage_client.bucket(bucket_name)<br>    for root, _, files in os.walk(model_dir):<br>        for file in files:<br>            local_file_path = os.path.join(root, file)<br>            gcs_file_path = os.path.relpath(local_file_path, model_dir)<br>            blob = bucket.blob(os.path.join(new_model, gcs_file_path))<br>            blob.upload_from_filename(local_file_path)<br><br>upload_to_gcs(BUCKET_DATA_NAME, save_model_path)<br>print(f&quot;Fine-tuned model {new_model} successfully uploaded to GCS.&quot;)</pre><p><strong>Dockerfile</strong>:</p><pre>FROM nvidia/cuda:12.6.2-runtime-ubuntu22.04<br>RUN apt-get update &amp;&amp; \<br>    apt-get -y --no-install-recommends install python3-dev gcc python3-pip git &amp;&amp; \<br>    rm -rf /var/lib/apt/lists/*<br>COPY requirements.txt .<br>RUN pip install --no-cache-dir --upgrade pip &amp;&amp; pip install --no-cache-dir -r requirements.txt<br>COPY finetuning.py .<br>ENV PYTHONUNBUFFERED 1<br>ENV MODEL_ID &quot;google/gemma-2-9b-it&quot;<br>ENV GCS_BUCKET &quot;finetuning-data-bucket&quot;<br>CMD [&quot;python3&quot;, &quot;finetuning.py&quot;]</pre><p><strong>requirements.txt</strong>:</p><pre>FROM nvidia/cuda:12.6.2-runtime-ubuntu22.04<br>RUN apt-get update &amp;&amp; \<br>    apt-get -y --no-install-recommends install python3-dev gcc python3-pip git &amp;&amp; \<br>    rm -rf /var/lib/apt/lists/*<br>COPY requirements.txt .<br>RUN pip install --no-cache-dir --upgrade pip &amp;&amp; pip install --no-cache-dir -r requirements.txt<br>COPY finetuning.py .<br>ENV PYTHONUNBUFFERED 1<br>ENV MODEL_ID &quot;google/gemma-2-9b-it&quot;<br>ENV GCS_BUCKET &quot;finetuning-data-bucket&quot;<br>CMD [&quot;python3&quot;, &quot;finetuning.py&quot;]</pre><p>This uses an Nvidia CUDA base image for GPU support, fine-tunes the model, and saves it to gs://&lt;BUCKET_DATA_NAME&gt;/fine_tuned_model.</p><p><em>What’s your favorite way to optimize fine-tuning? Let me know below!</em></p><h3>What’s Next?</h3><p>Your Airflow setup is ready, and the pipeline tasks are built! You’ve:</p><ul><li>Installed Airflow on GKE with Helm.</li><li>Configured the UI and connections.</li><li>Created containers for data download, preparation, and fine-tuning.</li></ul><p>In <strong>Part 4</strong>, we’ll define the Airflow DAG to run these tasks, deploy the fine-tuned model, and test it. That’s where the LLMOps pipeline comes full circle!</p><p>Explore the code in the or revisit the . Need a refresher? Check or .</p><p><em>What task would you add to this pipeline? Share your ideas!</em></p><h3>Quick Recap</h3><ul><li><strong>Airflow</strong>: Installed and configured on GKE.</li><li><strong>Data Download</strong>: Container to fetch Kaggle data.</li><li><strong>Data Preparation</strong>: Container to format data for fine-tuning.</li><li><strong>Fine-Tuning</strong>: Container to train Gemma-2–9b-it with LoRA.</li></ul><p>Try accessing the Airflow UI to poke around — it’s a sneak peek at Part 4’s action. The repo’s README has tips if you hit bumps.</p><p><em>Enjoying the series? Clap to show some love, and let’s talk Airflow in the comments!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=028970568f32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 2:Building an LLMOps Pipeline: Setting Up GKE for LLM Workloads]]></title>
            <link>https://medium.com/@sathiyakeerthi/part-2-building-an-llmops-pipeline-setting-up-gke-for-llm-workloads-e48b336a88ad?source=rss-90dfd0aea7b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e48b336a88ad</guid>
            <dc:creator><![CDATA[Sathiya Keerthi]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 02:09:24 GMT</pubDate>
            <atom:updated>2025-04-13T02:09:24.969Z</atom:updated>
            <content:encoded><![CDATA[<h4>Create a GPU-powered GKE cluster and storage to supercharge your LLMOps pipeline.</h4><p>Welcome back to our journey of building an LLMOps pipeline with <strong>Apache Airflow</strong> and <strong>Google Kubernetes Engine (GKE)</strong>! In , we set up Google Cloud, creating a project, enabling APIs, and laying the groundwork with a VPC and service account. Now, it’s time to build the infrastructure that’ll make our large language model (LLM) hum — a GKE cluster tailored for LLM workloads.</p><p>This is Part 2 of our four-part series, where we’ll transform raw cloud resources into a robust environment for fine-tuning and serving Gemma-2–9b-it on movie review data. Think of this as constructing the stage for our LLMOps performance: GKE for compute, Google Cloud Storage (GCS) for data, and Kubernetes manifests for orchestration. By the end, you’ll have a GPU-ready cluster waiting for Airflow’s magic touch.</p><p>New to GKE? Don’t worry — this guide is for LLM engineers, platform pros, and curious data scientists alike. All code lives in , and the gives the full scoop. Missed Part 1? Catch up .</p><p><em>What’s your favorite cloud infra setup for AI? Drop it in the comments!</em></p><h3>Why GKE for LLMs?</h3><p>GKE is Google’s managed Kubernetes service, perfect for LLMs because it handles cluster management while offering <strong>Nvidia L4 GPUs</strong> for training and inference. Pair that with GCS for scalable storage and Artifact Registry for containers, and you’ve got a dream team for LLMOps. Our pipeline will use this setup to process data, fine-tune a model, and deploy it — all orchestrated by Airflow in Part 3.</p><p>Today, we’ll:</p><ul><li>Create a GKE cluster with GPU node pools.</li><li>Set up GCS buckets for logs, DAGs, and data.</li><li>Configure Artifact Registry for container storage.</li><li>Define Kubernetes manifests for Airflow’s resources.</li><li>Assign IAM roles for secure access.</li></ul><p>Let’s turn our Google Cloud project into an LLM powerhouse!</p><h3>Step 1: Create a GKE Cluster</h3><p>Our pipeline needs compute muscle, so we’ll build a GKE cluster with <strong>Nvidia L4 GPUs</strong> — great for LLM tasks like fine-tuning. Run this in Cloud Shell, assuming you’ve set the environment variables from Part 1:</p><pre>gcloud container clusters create ${CLUSTER_NAME} \<br>  --zone ${REGION}-a \<br>  --num-nodes=4 \<br>  --network=llmops \<br>  --create-subnetwork name=llmops-subnet \<br>  --enable-ip-alias \<br>  --addons GcsFuseCsiDriver \<br>  --workload-pool=${DEVSHELL_PROJECT_ID}.svc.id.goog \<br>  --no-enable-insecure-kubelet-readonly-port \<br>  --service-account=${CLUSTER_SA}@${DEVSHELL_PROJECT_ID}.iam.gserviceaccount.com</pre><p>What’s this doing?</p><ul><li>Names the cluster <strong>llmops-airflow</strong> (from <strong>$CLUSTER_NAME</strong>).</li><li>Places it in <strong>us-central1-a</strong> (<strong>$REGION-a</strong>) for GPU availability.</li><li>Starts with 4 nodes and uses the <strong>llmops </strong>VPC from Part 1.</li><li>Enables <strong>GcsFuseCsiDriver</strong> to mount GCS buckets as volumes.</li><li>Uses <strong>workload identity</strong> for secure API access via the service account (<strong>sa-llmops-airflow)</strong>.</li></ul><p>This creates a general-purpose cluster. Now, add two node pools: one for <strong>training</strong> (1 L4 GPU) and one for <strong>inference</strong> (2 L4 GPUs):</p><pre>gcloud container node-pools create training \<br>  --accelerator type=nvidia-l4,count=1,gpu-driver-version=latest \<br>  --project=${PROJECT_ID} \<br>  --location=${REGION}-a \<br>  --node-locations=${REGION}-a \<br>  --cluster=${CLUSTER_NAME} \<br>  --machine-type=g2-standard-12 \<br>  --num-nodes=1<br>gcloud container node-pools create inference \<br>  --accelerator type=nvidia-l4,count=2,gpu-driver-version=latest \<br>  --project=${PROJECT_ID} \<br>  --location=${REGION}-a \<br>  --node-locations=${REGION}-a \<br>  --cluster=${CLUSTER_NAME} \<br>  --machine-type=g2-standard-24 \<br>  --num-nodes=1</pre><p>These node pools are tailored:</p><ul><li><strong>Training</strong>: <strong><em>g2-standard-12</em></strong> with 1 GPU for fine-tuning.</li><li><strong>Inference</strong>: <strong><em>g2-standard-24</em></strong> with 2 GPUs for serving predictions.</li></ul><p>Grab your cluster credentials to interact with Kubernetes:</p><pre>gcloud container clusters get-credentials ${CLUSTER_NAME} --location ${REGION}-a</pre><p><em>Ever worked with GPUs on GKE? What’s your setup like?</em></p><h3>Step 2: Set Up Google Cloud Storage</h3><p>Our pipeline needs storage for Airflow logs, DAGs (workflow definitions), and data. Create three GCS buckets:</p><pre>gcloud storage buckets create gs://${BUCKET_LOGS_NAME} --location=${REGION}<br>gcloud storage buckets create gs://${BUCKET_DAGS_NAME} --location=${REGION}<br>gcloud storage buckets create gs://${BUCKET_DATA_NAME} --location=${REGION}</pre><p>This creates:</p><ul><li><strong><em>llmops-airflow-logs-&lt;suffix&gt;</em></strong> for Airflow logs.</li><li><strong><em>llmops-airflow-dags-&lt;suffix&gt;</em></strong><em> </em>for DAG scripts.</li><li><strong><em>llmops-airflow-data-&lt;suffix&gt;</em></strong> for datasets and models.</li></ul><p>The <strong><em>$SUFFIX </em></strong>(random 4 characters) ensures unique names. All buckets are in <strong>us-central1</strong> for consistency with the cluster.</p><h3>Step 3: Configure Artifact Registry</h3><p>We’ll store Docker containers (e.g., for data processing and fine-tuning) in Artifact Registry. Set it up:</p><pre>gcloud artifacts repositories create ${REPO_NAME} \<br>  --repository-format=docker \<br>  --location=${REGION}<br>gcloud artifacts repositories add-iam-policy-binding ${REPO_NAME} \<br>  --member=serviceAccount:${CLUSTER_SA}@${DEVSHELL_PROJECT_ID}.iam.gserviceaccount.com \<br>  --role=roles/artifactregistry.reader \<br>  --location=${REGION}</pre><p>This:</p><ul><li>Creates <strong>llmops-airflow-repo</strong> for Docker images.</li><li>Grants the service account (<strong>sa-llmops-airflow</strong>) read access to pull images.</li></ul><h3>Step 4: Define Kubernetes Manifests</h3><p>Airflow needs Kubernetes resources like namespaces, persistent volumes, and service accounts to run on GKE. Let’s define them. First, create a manifests directory:</p><pre>mkdir manifests<br>cd manifests</pre><p>Now, create the YAML files. These are critical for Airflow’s storage and permissions — full versions are in the for easy copying.</p><p><strong>namespace.yaml</strong>:</p><pre>kind: Namespace<br>apiVersion: v1<br>metadata:<br>  name: airflow<br>  labels:<br>    name: airflow</pre><p>This creates an <strong>airflow</strong> namespace to isolate our resources.</p><p><strong>mlops-sa.yaml</strong> (renamed <strong>llmops-sa.yaml</strong>):</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>automountServiceAccountToken: true<br>metadata:<br>  name: airflow-llmops-sa<br>  namespace: airflow<br>  labels:<br>    tier: airflow</pre><p>This service account (<strong>airflow-llmops-sa</strong>) is for Airflow tasks.</p><p><strong>pv-dags.yaml</strong>:</p><pre>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: airflow-dags<br>spec:<br>  accessModes:<br>  - ReadWriteMany<br>  capacity:<br>    storage: 5Gi<br>  storageClassName: standard<br>  mountOptions:<br>    - implicit-dirs<br>  csi:<br>    driver: gcsfuse.csi.storage.gke.io<br>    volumeHandle: BUCKET_DAGS_NAME<br>    volumeAttributes:<br>      gcsfuseLoggingSeverity: warning</pre><p>Mounts the DAGs bucket as a 5Gi volume.</p><p><strong>pv-logs.yaml</strong>:</p><pre>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: airflow-logs<br>spec:<br>  accessModes:<br>  - ReadWriteMany<br>  capacity:<br>    storage: 100Gi<br>  storageClassName: standard<br>  mountOptions:<br>    - implicit-dirs<br>  csi:<br>    driver: gcsfuse.csi.storage.gke.io<br>    volumeHandle: BUCKET_LOGS_NAME<br>    volumeAttributes:<br>      gcsfuseLoggingSeverity: warning</pre><p>Mounts the logs bucket as a 100Gi volume.</p><p><strong>pvc-dags.yaml</strong>:</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: airflow-dags<br>  namespace: airflow<br>spec:<br>  accessModes:<br>  - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 5Gi<br>  volumeName: airflow-dags<br>  storageClassName: standard</pre><p>Claims the DAGs volume.</p><p><strong>pvc-logs.yaml</strong>:</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: airflow-logs<br>  namespace: airflow<br>spec:<br>  accessModes:<br>  - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 100Gi<br>  volumeName: airflow-logs<br>  storageClassName: standard</pre><p>Claims the logs volume.</p><p><strong>sa-role.yaml</strong>:</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  namespace: airflow<br>  name: airflow-deployment-role<br>rules:<br>- apiGroups: [&quot;apps&quot;]<br>  resources: [&quot;deployments&quot;]<br>  verbs: [&quot;create&quot;, &quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;services&quot;]<br>  verbs: [&quot;create&quot;, &quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;patch&quot;, &quot;update&quot;, &quot;delete&quot;]</pre><p>Defines permissions for Airflow to manage deployments and services.</p><p><strong>sa-rolebinding.yaml</strong>:</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: airflow-deployment-rolebinding<br>  namespace: airflow<br>subjects:<br>- kind: ServiceAccount<br>  name: airflow-worker<br>  namespace: airflow<br>roleRef:<br>  kind: Role<br>  name: airflow-deployment-role<br>  apiGroup: rbac.authorization.k8s.io</pre><p>Binds the role to Airflow’s worker service account.</p><p>Apply these manifests, updating bucket names:</p><pre>kubectl apply -f namespace.yaml<br>sed -i &quot;s/BUCKET_DAGS_NAME/${BUCKET_DAGS_NAME}/g&quot; pv-dags.yaml<br>sed -i &quot;s/BUCKET_LOGS_NAME/${BUCKET_LOGS_NAME}/g&quot; pv-logs.yaml<br>kubectl apply -f pv-dags.yaml<br>kubectl apply -f pv-logs.yaml<br>kubectl apply -f pvc-dags.yaml<br>kubectl apply -f pvc-logs.yaml<br>kubectl apply -f llmops-sa.yaml<br>kubectl apply -f sa-role.yaml<br>kubectl apply -f sa-rolebinding.yaml</pre><p>The <strong>sed </strong>commands replace placeholders with your bucket names (e.g., <strong>llmops-airflow-dags-&lt;suffix&gt;</strong>).</p><h3>Step 5: Assign IAM Roles</h3><p>To ensure Airflow can access storage, containers, and GKE resources, grant IAM roles using workload identity:</p><pre>gcloud projects add-iam-policy-binding ${DEVSHELL_PROJECT_ID} \<br>  --member &quot;principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${DEVSHELL_PROJECT_ID}.svc.id.goog/subject/ns/airflow/sa/airflow-scheduler&quot; \<br>  --role &quot;roles/storage.objectUser&quot;<br>gcloud projects add-iam-policy-binding ${DEVSHELL_PROJECT_ID} \<br>  --member &quot;principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${DEVSHELL_PROJECT_ID}.svc.id.goog/subject/ns/airflow/sa/airflow-triggerer&quot; \<br>  --role &quot;roles/storage.objectUser&quot;<br>gcloud projects add-iam-policy-binding ${DEVSHELL_PROJECT_ID} \<br>  --member &quot;principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${DEVSHELL_PROJECT_ID}.svc.id.goog/subject/ns/airflow/sa/airflow-worker&quot; \<br>  --role &quot;roles/storage.objectUser&quot;<br>gcloud projects add-iam-policy-binding ${DEVSHELL_PROJECT_ID} \<br>  --member &quot;principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${DEVSHELL_PROJECT_ID}.svc.id.goog/subject/ns/airflow/sa/airflow-worker&quot; \<br>  --role &quot;roles/container.developer&quot;<br>gcloud projects add-iam-policy-binding ${DEVSHELL_PROJECT_ID} \<br>  --member &quot;principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${DEVSHELL_PROJECT_ID}.svc.id.goog/subject/ns/airflow/sa/airflow-llmops-sa&quot; \<br>  --role &quot;roles/artifactregistry.reader&quot;<br>gcloud projects add-iam-policy-binding ${DEVSHELL_PROJECT_ID} \<br>  --member &quot;principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${DEVSHELL_PROJECT_ID}.svc.id.goog/subject/ns/airflow/sa/airflow-webserver&quot; \<br>  --role &quot;roles/storage.objectUser&quot;</pre><p>These grant:</p><ul><li><strong>storage.objectUser</strong> to scheduler, triggerer, worker, and webserver for GCS access.</li><li><strong>container.developer</strong> to worker for Kubernetes operations.</li><li><strong>artifactregistry.reader</strong> to <strong>airflow-llmops-sa</strong> for pulling containers.</li></ul><h3>What’s Next?</h3><p>Your GKE cluster is live, buzzing with GPUs and backed by storage and containers! You’ve built:</p><ul><li>A GKE cluster with training and inference node pools.</li><li>GCS buckets for logs, DAGs, and data.</li><li>An Artifact Registry for Docker images.</li><li>Kubernetes manifests for Airflow.</li><li>IAM roles for secure access.</li></ul><p>In <strong>Part 3</strong>, we’ll install Airflow on this cluster and build the pipeline tasks — data downloading, preparation, and fine-tuning. It’s where the LLMOps action starts heating up!</p><p>Check the for all manifests or revisit the for the roadmap. Curious about Part 1? It’s .</p><p><em>What’s your biggest GKE win so far? Share below!</em></p><h3>Quick Recap</h3><ul><li><strong>Cluster</strong>: GKE with Nvidia L4 GPUs, ready for LLMs.</li><li><strong>Storage</strong>: Three GCS buckets for pipeline needs.</li><li><strong>Registry</strong>: Artifact Registry for containers.</li><li><strong>Kubernetes</strong>: Manifests for Airflow’s namespace, volumes, and roles.</li><li><strong>IAM</strong>: Permissions set for secure operations.</li></ul><p>Run these commands in Cloud Shell to see your cluster come alive. The repo’s README has tips if you hit snags.</p><p><em>Enjoyed this setup? Clap to show some love, and let’s talk GKE in the comments!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e48b336a88ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1: Building an LLMOps Pipeline: Getting Started with Google Cloud]]></title>
            <link>https://medium.com/@sathiyakeerthi/part-1-building-an-llmops-pipeline-getting-started-with-google-cloud-f826889a3920?source=rss-90dfd0aea7b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f826889a3920</guid>
            <dc:creator><![CDATA[Sathiya Keerthi]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 02:08:37 GMT</pubDate>
            <atom:updated>2025-04-13T02:08:37.555Z</atom:updated>
            <content:encoded><![CDATA[<h4>Kick off your LLMOps journey by setting up Google Cloud for a scalable LLM pipeline.</h4><h3>Welcome to the World of LLMOps</h3><p>Large language models (LLMs) are game-changers — think sentiment analysis that nails movie reviews or chatbots that feel almost human. But here’s the rub: deploying LLMs isn’t just about training them. It’s about automating the entire process, from data ingestion to production inference, so your model stays sharp and scalable. That’s where <strong>LLMOps</strong> (Large Language Model Operations) swoops in, blending DevOps discipline with LLM magic.</p><p>This is Part 1 of a four-part series on building an LLMOps pipeline using <strong>Apache Airflow</strong> and <strong>Google Kubernetes Engine (GKE)</strong>. We’ll transform a movie review dataset into a fine-tuned Gemma-2–9b-it model, ready to predict sentiments like “POSITIVE” or “NEGATIVE.” Today, we’re laying the foundation: setting up Google Cloud to power our pipeline.</p><p>Never worked with GKE or Airflow before? No sweat. This guide is for LLM engineers, platform pros, data scientists — anyone curious about productionizing LLMs. All code is in , so you can follow along hands-on. Want the big picture? Check out the .</p><p><em>What’s your dream LLM project? Share in the comments!</em></p><h3>Why Google Cloud for LLMOps?</h3><p>Google Cloud is a beast for LLMs. It offers <strong>Cloud Shell</strong> for easy command-line access, <strong>GKE</strong> for GPU-powered Kubernetes, and <strong>Google Cloud Storage (GCS)</strong> for scalable data handling. Plus, new users get a $300 free trial, so you can experiment without breaking the bank. Our pipeline will use these tools to download data, fine-tune a model, and serve it, all orchestrated by Airflow.</p><p>In this part, we’ll:</p><ul><li>Create a Google Cloud project and verify access.</li><li>Set up API tokens for Kaggle and Hugging Face.</li><li>Configure environment variables, APIs, a VPC, and a service account.</li></ul><p>By the end, you’ll have a cloud environment ready for GKE. Let’s get started!</p><h3>Step 1: Fire Up Your Google Cloud Project</h3><p>First, you need a Google Cloud project — a container for all your resources. Head to the <a href="http://console.cloud.google.com/"><strong><em>Google Cloud Console</em></strong></a> and click “New Project.” Give it a name (e.g., “llmops-pipeline”) and enable billing. Don’t worry — Google’s free trial covers most of what we’ll do.</p><p>Once your project is live, open <strong>Cloud Shell</strong> by clicking the terminal icon in the Console’s top-right corner. Cloud Shell is a browser-based Linux environment with <strong>gcloud</strong> preinstalled — perfect for our setup. Verify you’re authenticated and in the right project:</p><p>This lists your active account, like <strong>your-email@domain.com</strong>. Next, check your project ID:</p><pre>gcloud config list project</pre><p>If it’s not your new project, set it:</p><pre>gcloud config set project YOUR_PROJECT_ID</pre><p>Replace <strong>YOUR_PROJECT_ID</strong> with the ID from the Console (e.g.,<strong> llmops-pipeline-123456</strong>). This ensures all commands target the right project.</p><p><em>Pro tip</em>: Bookmark your project in the Console to switch quickly. Got a favorite Cloud Shell trick? Drop it below!</p><h3>Step 2: Grab API Tokens</h3><p>Our pipeline pulls movie reviews from <a href="https://www.kaggle.com/">Kaggle</a> and the Gemma-2–9b-it model from <a href="https://huggingface.co/">Hugging Face</a>. You’ll need API tokens for both.</p><p>For <strong>Kaggle</strong>:</p><ul><li>Sign up or log in.</li><li>Go to your profile → “Settings” → “API” → “Create New Token.”</li><li>Download <strong>kaggle.json</strong>. It contains your <strong>username </strong>and <strong>key </strong>— save these securely.</li></ul><p>For <strong>Hugging Face</strong>:</p><ul><li>Create an account or log in.</li><li>Navigate to <a href="https://huggingface.co/settings/tokens">Settings/Tokens</a> and generate a <strong>Read</strong> token.</li><li>Accept the license for Gemma-2–9b-it at <a href="https://huggingface.co/google/gemma-2-9b-it"><strong>this link</strong></a>.</li></ul><p>Store these tokens somewhere safe (we’ll use them in Part 3). They’re your keys to data and models.</p><p><em>Ever hit a snag with API tokens? What’s your workaround?</em></p><h3>Step 3: Set Environment Variables</h3><p>To keep commands clean, let’s define environment variables. These name our resources — like buckets and clusters — ensuring consistency. Run this in Cloud Shell:</p><pre>export CODELAB_PREFIX=llmops-airflow<br>export PROJECT_NUMBER=$(gcloud projects list --filter=&quot;${DEVSHELL_PROJECT_ID}&quot; --format=&quot;value(PROJECT_NUMBER)&quot;)<br>SUFFIX=$(echo $RANDOM | md5sum | head -c 4; echo;)<br>export CLUSTER_NAME=${CODELAB_PREFIX}<br>export CLUSTER_SA=sa-${CODELAB_PREFIX}<br>export BUCKET_LOGS_NAME=${CODELAB_PREFIX}-logs-${SUFFIX}<br>export BUCKET_DAGS_NAME=${CODELAB_PREFIX}-dags-${SUFFIX}<br>export BUCKET_DATA_NAME=${CODELAB_PREFIX}-data-${SUFFIX}<br>export REPO_NAME=${CODELAB_PREFIX}-repo<br>export REGION=us-central1<br>export PROJECT_ID=${DEVSHELL_PROJECT_ID}</pre><p>What’s happening here?</p><ul><li><strong>LLMOPS_PREFIX </strong>sets a consistent naming theme (<strong>llmops-airflow</strong>).</li><li><strong>PROJECT_NUMBER</strong> grabs your project’s unique number.</li><li><strong>SUFFIX</strong> adds randomness to bucket names to avoid collisions.</li><li>Others define names for the GKE cluster, service account, buckets, and Artifact Registry.</li></ul><p>The <strong>REGION (us-central1)</strong> is chosen for Nvidia L4 GPU availability — check Google Cloud’s <a href="https://cloud.google.com/compute/docs/regions-zones"><strong>region docs</strong></a> if you prefer another.</p><h3>Step 4: Enable Google Cloud APIs</h3><p>Our pipeline relies on GKE, Cloud Build, Artifact Registry, and GCS. Enable their APIs:</p><pre>gcloud services enable \<br>  container.googleapis.com \<br>  cloudbuild.googleapis.com \<br>  artifactregistry.googleapis.com \<br>  storage.googleapis.com</pre><p>This activates:</p><ul><li><strong>container.googleapis.com</strong> for GKE clusters.</li><li><strong>cloudbuild.googleapis.com</strong> for building containers.</li><li><strong>artifactregistry.googleapis.com</strong> for storing containers.</li><li><strong>storage.googleapis.com</strong> for GCS buckets.</li></ul><p>Run this once per project — it’s like flipping the switch for your cloud tools.</p><h3>Step 5: Create a VPC and Service Account</h3><p>To isolate our GKE cluster’s network, create a Virtual Private Cloud (VPC):</p><pre>gcloud compute networks create llmops --subnet-mode=auto</pre><p>The <strong>llmops</strong> VPC uses automatic subnet creation for simplicity. This keeps our cluster’s traffic secure and organized.</p><p>Next, create a service account for the GKE cluster to interact with Google Cloud:</p><pre>gcloud iam service-accounts create ${CLUSTER_SA} --display-name=&quot;SA for ${CLUSTER_NAME}&quot;<br>gcloud projects add-iam-policy-binding ${DEVSHELL_PROJECT_ID} \<br>  --member &quot;serviceAccount:${CLUSTER_SA}@${DEVSHELL_PROJECT_ID}.iam.gserviceaccount.com&quot; \<br>  --role roles/container.defaultNodeServiceAccount</pre><p>This does two things:</p><ul><li>Creates <strong>sa-llmops-airflow</strong> with a descriptive name.</li><li>Grants it the <strong>container.defaultNodeServiceAccount</strong> role, allowing basic GKE node operations.</li></ul><p>Think of the service account as your cluster’s ID card — it proves it’s allowed to access cloud resources.</p><p><em>What’s your go-to way to manage cloud networking? Share your tips!</em></p><h3>What’s Next?</h3><p>You’ve set up Google Cloud like a pro! Your project is live, APIs are enabled, and the groundwork — environment variables, VPC, and service account — is ready. In <strong>Part 2</strong>, we’ll build the GKE cluster, add GPU node pools, and set up storage, turning this foundation into a powerhouse for LLM workloads.</p><p>Can’t wait? Peek at the code in the or review the for the full roadmap. Part 2 will bring us closer to running our LLM pipeline with Airflow.</p><h3>Quick Recap</h3><ul><li><strong>Project</strong>: Created and verified in Google Cloud.</li><li><strong>Tokens</strong>: Secured for Kaggle and Hugging Face.</li><li><strong>Environment</strong>: Variables set for consistency.</li><li><strong>APIs</strong>: Enabled for GKE, Cloud Build, and more.</li><li><strong>Network</strong>: VPC created for isolation.</li><li><strong>Service Account</strong>: Ready for GKE access.</li></ul><p>This setup is the first step to a scalable LLMOps pipeline. Try running the commands in Cloud Shell to see it come alive. If you hit bumps, the repo’s README has troubleshooting tips.</p><p><em>What did you think of this setup? Clap if it clicked, and let me know your thoughts below!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f826889a3920" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an LLMOps Pipeline with Airflow and GKE]]></title>
            <link>https://medium.com/@sathiyakeerthi/introduction-page-welcome-to-building-an-llmops-pipeline-with-airflow-and-gke-a76cf48c4b72?source=rss-90dfd0aea7b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a76cf48c4b72</guid>
            <dc:creator><![CDATA[Sathiya Keerthi]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 02:08:06 GMT</pubDate>
            <atom:updated>2025-04-23T01:21:26.346Z</atom:updated>
            <content:encoded><![CDATA[<h4>A four-part series to master LLMOps, from Google Cloud setup to deploying a fine-tuned LLM.</h4><h3>The Promise of LLMOps</h3><p>Large language models (LLMs) like Gemma-2–9b-it are transforming AI, powering everything from chatbots to sentiment analysis. But here’s the catch: training an LLM is only half the battle. Getting it into production — keeping it updated, scalable, and reliable — is where things get messy. Enter <strong>LLMOps</strong>, the art of blending DevOps with LLM workflows to automate the entire lifecycle, from data processing to deployment.</p><p>This four-part series is your hands-on guide to building an LLMOps pipeline using <strong>Apache Airflow</strong> on <strong>Google Kubernetes Engine (GKE)</strong>. We’ll take a movie review dataset, fine-tune an LLM to analyze sentiments, and serve it in production, all orchestrated with Airflow’s powerful Directed Acyclic Graphs (DAGs). Whether you’re an LLM engineer dreaming of streamlined models or a platform pro wrangling cloud infrastructure, this series is for you.</p><blockquote><em>“LLMOps isn’t just about automation — it’s about making AI practical and scalable.”<br> Ready to simplify your LLM deployments? Let’s explore what’s ahead.</em></blockquote><h3>Why This Series?</h3><p>LLMs are resource-hungry beasts, and managing them manually is a recipe for chaos. Airflow and GKE offer a robust combo: Airflow schedules and monitors tasks, while GKE provides GPU-powered compute and Kubernetes scalability. Together, they create a pipeline that’s repeatable, efficient, and production-ready.</p><p>In this series, we’ll:</p><ul><li>Set up a Google Cloud environment tailored for LLMs.</li><li>Build a GKE cluster with GPU nodes for training and inference.</li><li>Use Airflow to orchestrate data processing, fine-tuning, and deployment.</li><li>Deploy a fine-tuned LLM with vLLM for high-performance serving.</li></ul><p>No prior Airflow or GKE expertise is needed — just a curiosity for LLMs and a Google Cloud account (new users get a $300 free trial). All code lives in — clone it to follow along!</p><p><em>What’s your biggest hurdle with LLM deployments? Share in the comments!</em></p><h3>What’s in Each Part?</h3><p>This series breaks down the LLMOps pipeline into four digestible steps, each building on the last. Here’s what you’ll learn:</p><h4><a href="https://medium.com/@sathiyakeerthi/part-1-building-an-llmops-pipeline-getting-started-with-google-cloud-f826889a3920">Part 1: Getting Started with Google Cloud</a></h4><p>Kick off by setting up your Google Cloud project. You’ll configure Cloud Shell, enable APIs, and create a VPC and service account to prepare for GKE. This part is all about laying a solid foundation for your LLM workloads.<br> (Link placeholder — to be updated when published)</p><h4><a href="https://medium.com/@sathiyakeerthi/part-2-building-an-llmops-pipeline-setting-up-gke-for-llm-workloads-e48b336a88ad">Part 2: Setting Up GKE for LLM Workloads</a></h4><p>Dive into infrastructure: create a GKE cluster with GPU-enabled node pools, set up Google Cloud Storage buckets, and define Kubernetes resources. By the end, you’ll have a scalable environment ready for LLM tasks.</p><h4><a href="https://medium.com/@sathiyakeerthi/part-3-building-an-llmops-pipeline-orchestrating-tasks-with-airflow-028970568f32">Part 3: Orchestrating Tasks with Airflow</a></h4><p>Meet Airflow, your pipeline’s conductor. You’ll install it on GKE, configure the UI, and build containers for downloading data, preparing it, and fine-tuning the LLM. This part brings the pipeline’s pieces together.</p><h4><a href="https://medium.com/@sathiyakeerthi/part-4-building-an-llmops-pipeline-fine-tuning-and-serving-your-llm-2fb5e60d8ecb">Part 4: Fine-Tuning and Serving Your LLM</a></h4><p>Tie it all up with an Airflow DAG to run the pipeline. You’ll fine-tune Gemma-2–9b-it, deploy it with vLLM, and test its sentiment analysis. Plus, get tips for taking it to production.</p><p>Each part is packed with code, explanations, and tips, designed for both LLM engineers and platform teams. Expect to spend 6–9 minutes per article, with all the tools you need to build it yourself.</p><h3>Who’s This For?</h3><p>This series is crafted for:</p><ul><li><strong>LLM Engineers</strong> eager to automate model training and deployment.</li><li><strong>Platform Engineers</strong> building infrastructure for AI workloads.</li><li><strong>Data Scientists</strong> curious about productionizing LLMs.</li><li><strong>DevOps Pros</strong> integrating Kubernetes with AI pipelines.</li></ul><p>If you’ve ever struggled to move an LLM from notebook to production — or wondered how to make it happen — this series is your blueprint.</p><h3>What You’ll Need</h3><p>To follow along, grab these:</p><ul><li>A <a href="http://console.cloud.google.com/">Google Cloud account</a> with billing enabled.</li><li>API tokens from <a href="https://www.kaggle.com/">Kaggle</a> and <a href="https://huggingface.co/">Hugging Face</a> — details in Part 1.</li><li>Basic command-line skills (don’t worry, we’ll guide you).</li></ul><p>The has every script, Dockerfile, and YAML file, so you can focus on learning, not copying code.</p><p><em>Got your cloud account ready? What’s your first step?</em></p><h3>Why Fine-Tuning?</h3><p>We’ll fine-tune Gemma-2–9b-it using Low-Rank Adaptation (LoRA) instead of Retrieval-Augmented Generation (RAG). Fine-tuning creates a specialized model for our dataset, making inference fast and lightweight — perfect for static movie reviews. RAG suits dynamic data but adds complexity. Curious why we chose this? Part 4 dives deeper.</p><h3>Let’s Get Started</h3><p>Start with <strong>Part 1</strong> to set up Google Cloud, then follow the journey to a fully deployed LLM. I’ll be here in the comments to answer questions, so don’t hesitate to ask. Clap if you’re excited, and let’s make LLMOps work for you!</p><p><em>What LLM project are you dreaming of? Tell me below!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a76cf48c4b72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy a Retrieval-Augmented Generation (RAG) Application on GKE: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@sathiyakeerthi/how-to-deploy-a-retrieval-augmented-generation-rag-application-on-gke-a-step-by-step-guide-b92dd03f62ff?source=rss-90dfd0aea7b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b92dd03f62ff</guid>
            <dc:creator><![CDATA[Sathiya Keerthi]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 21:34:52 GMT</pubDate>
            <atom:updated>2025-04-08T02:27:43.405Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine teaching an AI to answer questions about Netflix shows with pinpoint accuracy. That’s the power of Retrieval-Augmented Generation (RAG) — a technique that supercharges large language models (LLMs) by pairing them with a searchable knowledge base.</p><p>In this article, I’ll walk you through deploying a RAG application on <strong>Google Kubernetes Engine (GKE)</strong> — step-by-step, like a classroom lab. By the end, you’ll have a working chat interface powered by Gemma 7B and a <strong>Netflix dataset</strong>.</p><p>Let’s get started!</p><h3>RAG makes LLMs smarter by giving them <em>real-time context</em> from a dataset. Instead of retraining the model (which is costly and slow), RAG uses a <strong>vector database</strong> to retrieve relevant info — like Netflix show descriptions — and feeds it to the LLM.</h3><p><strong>Why it matters:</strong> It’s perfect for domain-specific knowledge or fast-changing content.</p><p><strong>Our setup:</strong></p><ul><li><strong>AI Model</strong>: Gemma-7b for answering questions</li><li><strong>Vector Database</strong>: Cloud SQL with pgvector for storing embeddings.</li><li><strong>Tools</strong>: A Ray cluster, Jupyter notebook, and chat UI — all on GKE.</li></ul><h3>Step 1: Gather Your Tools</h3><p>Before we begin, install the essentials:</p><ul><li>kubectl: Control Kubernetes clusters</li><li>Terraform: Automate infrastructure setup</li><li>Helm: Manage Kubernetes apps</li><li>gcloud: Work with Google Cloud</li></ul><h3>Step 2: Understand the Architecture</h3><p>Here’s how everything fits together:</p><ol><li>A <strong>Jupyter notebook</strong> processes the Netflix dataset from Kaggle.</li><li>A <strong>Ray cluster</strong> turns data into vector embeddings and stores them in pgvector.</li><li>A <strong>chat UI</strong> retrieves context, adds it to your question, and sends it to Gemma-7b.</li></ol><h3>🧱 Architecture Overview</h3><p>This diagram represents a <strong>production-grade RAG application</strong> deployed on GKE Autopilot with the following two key subsystems:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*E2JLd8Dc4WMnhqKNbqjDlw.png" /></figure><h3>✅ Serving Subsystem</h3><p>The <strong>serving subsystem</strong> handles real-time, user-facing request and response flow:</p><ul><li>A <strong>frontend server</strong> accepts natural language input through a chat interface.</li><li>The <strong>LangChain</strong> framework converts the input to an embedding and queries a vector database for relevant data (called grounding data).</li><li>A <strong>contextualized prompt</strong> is created by combining the original query with the retrieved data.</li></ul><p>The prompt is sent to the <strong>inference server</strong> (running Hugging Face TGI with Gemma-7B). Ref:<a href="https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/deploy-gemma-from-gcs-on-vertex-ai/vertex-notebook.ipynb"> Deploy Gemma from GCS on Vertex AI</a></p><ul><li>The model response is filtered by a <strong>Responsible AI (RAI) service</strong>.</li><li>The final output is returned to the user.</li></ul><p>You can monitor this entire flow with <strong>Cloud Logging</strong>, <strong>Cloud Monitoring</strong>, and analyze model responses in <strong>BigQuery</strong>.</p><h3>📥 Embedding Subsystem</h3><p>The <strong>embedding subsystem</strong> is responsible for powering RAG’s retrieval capability:</p><ol><li>Data is uploaded to <strong>Cloud Storage</strong> by users or services (can be automated with Pub/Sub triggers).</li><li>The <strong>embedding service</strong>, deployed on GKE, is notified and:</li></ol><ul><li>Reads data via Cloud Storage FUSE CSI driver.</li><li>Uses <strong>Ray Data</strong> to preprocess, chunk, and prepare data.</li><li>Generates embeddings using a model like intfloat/multilingual-e5-small</li><li>Stores embeddings in <strong>Cloud SQL with pgvector</strong> (used for semantic search).</li></ul><p>The vector database supports <strong>semantic search</strong> based on embedding similarity, allowing the system to retrieve answers based on <strong>meaning</strong>, not just keywords.</p><blockquote><em>Note: The networking details (like VPCs, subnetworks, and load balancers) are not shown in the diagram. You can refer to GKE Networking Best Practices for production-ready configuration.</em></blockquote><h3>Step 3: Provision Infrastructure with Terraform</h3><p>Terraform is our blueprint for spinning up GKE, databases, and networking.</p><p>You can explore the full code, Terraform configs, and notebooks here:<br> 👉 <strong>GitHub Repository</strong>: <a href="https://github.com/msathiyakeerthi/rag-on-gke">https://github.com/msathiyakeerthi/rag-on-gke</a></p><h3>Get the Code:</h3><pre>git clone https://github.com/msathiyakeerthi/rag-on-gke.git</pre><h3>Configure:</h3><p>Edit workloads.tfvars and set:</p><ul><li>project_id = &quot;&lt;your-gcp-project-id&gt;&quot;</li><li>location = &quot;us-central1&quot;</li><li>cluster_name = &quot;rag-cluster&quot;</li><li>bucket_name = &quot;my-rag-bucket-123&quot;</li></ul><h3>Deploy:</h3><pre>terraform init<br>terraform apply --var-file workloads.tfvars</pre><blockquote><em>Confirm with </em><em>yes.</em></blockquote><blockquote><em>Setup takes ~10 minutes. Don’t delete </em><em>terraform.tfstate—it tracks infrastructure.</em></blockquote><h3>Step 4: Process Netflix Data</h3><h3>Connect to GKE:</h3><pre>export NAMESPACE=rag<br>export CLUSTER_NAME=rag-cluster<br>export CLUSTER_LOCATION=us-east4<br>gcloud container clusters get-credentials $CLUSTER_NAME --location=$CLUSTER_LOCATION</pre><h3>Launch JupyterHub:</h3><pre>kubectl port-forward service/proxy-public -n $NAMESPACE 8081:80 &amp;</pre><p>Open <a href="http://localhost:8081/">http://localhost:8081</a>.<br> Login using admin and the password from the terraform output jupyterhub_password.</p><h3>Load Notebook:</h3><p>In JupyterHub:<br> <strong>File &gt; Open From URL</strong>, then paste:</p><pre>https://raw.githubusercontent.com/GoogleCloudPlatform/ai-on-gke/main/applications/rag/example_notebooks/rag-kaggle-ray-sql-interactive.ipynb</pre><h3>Add Kaggle API Key:</h3><p>Upload kaggle.json, and update the first notebook cell with your username and key.</p><h3>Run All Cells:</h3><p>This downloads the Netflix dataset, generates embeddings with Ray, and saves them to Cloud SQL.</p><blockquote><em>Expect ~10 minutes runtime.</em></blockquote><h3>Step 5: Launch the Chat UI</h3><pre>kubectl port-forward service/rag-frontend -n $NAMESPACE 8080:8080 &amp;</pre><p>Then, go to <a href="http://localhost:8080/">http://localhost:8080</a><br> Ask: “<strong>What’s a good sci-fi show on Netflix?</strong>” and watch the magic unfold.</p><h3>Step 6: Clean Up</h3><p>To avoid cloud charges:</p><pre>terraform destroy --var-file workloads.tfvars</pre><blockquote><em>Confirm with </em><em>yes. If the VPC network sticks around, delete it manually via GCP Console.</em></blockquote><h3>Step 7: Troubleshooting Tips</h3><ul><li><strong>Notebook stuck?</strong></li></ul><pre>kubectl port-forward -n $NAMESPACE service/ray-cluster-kuberay-head-svc 8265:8265</pre><ul><li>Then open <a href="http://localhost:8265/">http://localhost:8265</a> to debug Ray.</li><li><strong>Model not loading?</strong><br> Make sure you have <strong>L4 GPU quota</strong> in your region.</li><li><strong>HF gated model?</strong><br> Add your Hugging Face access token as a Kubernetes secret (details in the GitHub tutorial).</li></ul><h3>Why This Project Matters</h3><p>This project showcases a <strong>modern AI deployment stack</strong>:</p><ul><li><strong>Infrastructure-as-Code</strong> with Terraform</li><li><strong>Data preprocessing</strong> and embedding with Ray</li><li><strong>Real-time inference</strong> with Gemma + pgvector</li><li><strong>Scalability</strong> with GKE</li></ul><p>You can easily <strong>swap the Netflix dataset</strong> for your own — internal docs, product catalogs, or support tickets — and adapt this to enterprise needs.</p><h3>Let’s Chat</h3><p>Have you explored RAG or GKE for AI workloads?<br> Drop your thoughts or questions in the comments. I’d love to hear how you’re using these tools!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b92dd03f62ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ How DeepSeek-V3 Picks Perfect Experts for Every Word-Ditch Softmax]]></title>
            <link>https://medium.com/@sathiyakeerthi/how-deepseek-v3-picks-perfect-experts-for-every-word-ditch-softmax-3dfa20e69e8d?source=rss-90dfd0aea7b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dfa20e69e8d</guid>
            <dc:creator><![CDATA[Sathiya Keerthi]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 23:23:10 GMT</pubDate>
            <atom:updated>2025-01-22T02:31:33.106Z</atom:updated>
            <content:encoded><![CDATA[<p>Training a 671B-parameter AI model sounds impractical — until you realize <strong>97% of its neurons stay dormant for each token</strong>. Meet DeepSeek-V3’s <strong>dynamic hierarchical gating</strong> — the brain behind its surgical efficiency. Let’s dissect this architectural marvel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*UYqnFDu6d6WnrHKb0xRp8w.png" /></figure><p>DeepSeekMoE 16B is a Mixture-of-Experts (MoE) language model with 16.4B parameters. It employs an innovative MoE architecture, which involves two principal strategies: fine-grained expert segmentation and shared experts isolation. It is trained from scratch on 2T English and Chinese tokens, and exhibits comparable performance with DeekSeek 7B and LLaMA2 7B, with only about 40% of computations. For research purposes, we release the model checkpoints of DeepSeekMoE 16B Base and DeepSeekMoE 16B Chat to the public, which can be deployed on a single GPU with 40GB of memory without the need for quantization. The model code file can be found <a href="https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/main/modeling_deepseek.py">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtdEFL4uhqG_cud0KdvC4A.png" /></figure><h3>❌ The Problem With Traditional MoE: Why Softmax Fails</h3><p>Most MoE models use a <strong>flawed beauty pageant approach</strong>:</p><pre># Traditional MoE gating (simplified)<br>scores = softmax(linear(x))  # Forces experts to compete<br>top_experts = scores.topk(k=2)</pre><p>This creates three critical issues:</p><ol><li><strong>Expert starvation</strong>: Only popular experts get selected</li><li><strong>Training complexity</strong>: Requires auxiliary losses for balance</li><li><strong>Hardware inefficiency</strong>: 40% GPU memory wasted on inactive experts</li></ol><p>DeepSeek-V3’s solution? <strong>Replace beauty pageants with talent shows</strong> 🎭</p><h3>🔍 The 3-Step Expert Selection Dance</h3><h3>1️⃣ Sigmoid Scoring: The Open Audition</h3><p><strong>File</strong>: <a href="https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py#L428-L434">model.py#L428-L434</a></p><pre># model.py (Gate class)<br>def forward(self, x):<br>    scores = torch.matmul(x, self.weight.t())  # [batch, 256 experts]<br>    scores = scores.sigmoid()  # No forced competition!</pre><p><strong>Why it matters</strong>:</p><ul><li>Each expert gets <strong>independent score</strong> (0–1)</li><li>Enables <strong>multi-expert activation</strong> for complex tokens</li><li><em>Example</em>: “Quantum” token activates both physics and math experts</li></ul><h3>2️⃣ Group Filtering: The VIP Section</h3><p><strong>Config</strong>: <a href="https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_671B.json#L14">config_671B.json#L14</a></p><pre>{<br>    &quot;n_expert_groups&quot;: 8,<br>    &quot;n_limited_groups&quot;: 4<br>}</pre><pre># model.py (Group selection logic)<br>group_scores = scores.view(batch_size, 8, 32).max(dim=-1)<br>top_groups = group_scores.topk(k=4)  # Select &quot;Physics&quot;, &quot;Math&quot;, etc.</pre><p><strong>Key innovation</strong>:</p><ul><li>Prevents domain monopolization</li><li>Ensures <strong>diverse expertise</strong> per token</li><li>Reduces candidate experts from 256 → 32</li></ul><h3>3️⃣ Expert Selection: The Final Cut</h3><p><strong>File</strong>: <a href="https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py#L443-L449">model.py#L443-L449</a></p><pre>filtered_scores = scores * group_mask  # Zero non-VIP groups<br>top_experts = filtered_scores.topk(k=8)<br>weights = normalize(top_experts) * 2.5  # Learned scaling</pre><p><strong>Result</strong>:</p><ul><li>8 experts selected from 32 candidates</li><li>37B params activated/token vs 240B in GPT-4</li></ul><h3>⚖️ Load Balancing Magic (No Training Hacks)</h3><p>Traditional MoE requires complex loss terms like:</p><pre>aux_loss = expert_usage.var()  # Force uniform usage</pre><p>DeepSeek-V3 achieves balance through:</p><ol><li><strong>Natural group rotation</strong>: Experts cycle through active groups</li><li><strong>Sigmoid flexibility</strong>: Underused experts gain priority over time</li><li><strong>Hardware-aware sharding</strong>: Experts distributed across GPUs</li></ol><h4>🌍 Real-World Example</h4><p>For the input <em>“Schrödinger’s cat in quantum superposition”</em>:</p><ol><li><strong>Groups selected</strong>: Physics (max score), Philosophy, Math</li><li><strong>Experts activated</strong>: Quantum mechanics specialist Metaphysics parserProbability calculator</li><li><strong>Result</strong>: Perfect blend of domain knowledge without wasteful computation</li></ol><h4>💡 Why This Changes Everything</h4><ul><li><strong>5x cheaper inference</strong> vs dense models with similar capability</li><li><strong>Trillion-parameter future</strong> becomes practic</li><li><strong>Democratizes large-scale AI</strong> through efficient open-source model</li></ul><p><strong>🤔 Food for Thought</strong></p><ul><li>Could hierarchical gating work for multi-modal models?</li><li>Will sparse expert networks become the default for LLMs?</li><li>How low can we push activated parameters? (Current record: 5.5%)</li></ul><h3>📊 Head-to-Head Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n3WRAcsqYpmsDjNW_T6g-Q.png" /></figure><h3>💥 Why This Matters</h3><p>DeepSeek-V3’s approach isn’t just technically superior — it <strong>democratizes large-scale AI</strong>:</p><ol><li><strong>Startups</strong> can now access GPT-4-level capabilities without cloud lock-in</li><li><strong>Enterprises</strong> gain control over expert specialization (e.g., healthcare vs fintech groups)</li><li><strong>Researchers</strong> get a blueprint for sustainable trillion-parameter models</li></ol><p><em>P.S. To the engineers — yes, the code is public. Linked the GitHub repo in comments! 👇</em></p><p><strong>Like</strong> if you want a deep dive into implementing this with PyTorch!<br><strong>Repost</strong> to spread the word about efficient AI design!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dfa20e69e8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>