Use vLLM on GKE to run inference with Qwen3

Access Qwen3 by using Hugging Face

To use Hugging Face to access Qwen3, follow these steps:

1. Sign in to Hugging Face
2. Create a Hugging Face read access token. Click Your Profile > Settings > Access Tokens > +Create new token.
3. Specify a name of your choice for the token and then select a role. The minimum role permission level that you can select for this tutorial is Read.
4. Select Create token.
5. Copy and save the generated token to your clipboard. You use it later in this tutorial.

Prepare your environment

To prepare your environment, set the default environment variables:

export PROJECT_ID="YOUR_PROJECT_ID"
export RESERVATION_NAME="YOUR_RESERVATION_NAME"
export REGION="YOUR_REGION"
export CLUSTER_NAME="YOUR_CLUSTER_NAME"
export GCS_BUCKET_NAME="YOUR_GCS_BUCKET"
export HUGGING_FACE_TOKEN="YOUR_HF_TOKEN"
export NETWORK="YOUR_NETWORK_NAME"
export SUBNETWORK="YOUR_SUBNETWORK_NAME"

gcloud config set project "${PROJECT_ID}"
gcloud config set billing/quota_project "${PROJECT_ID}"

Replace the following:

• YOUR_PROJECT_ID: the ID of the Google Cloud project where you want to create the GKE cluster.

• YOUR_RESERVATION_NAME: the name of the reservation that you want to use to create your GKE cluster. Based on the project in which the reservation exists, specify one of the following values:

  • The reservation exists in your project: RESERVATION_NAME

  • The reservation exists in a different project, and your project can use the reservation: projects/RESERVATION_PROJECT_ID/reservations/RESERVATION_NAME

• YOUR_REGION: the region where you want to create your GKE cluster. You can only create the cluster in the region where your reservation exists.

• YOUR_CLUSTER_NAME: the name of the GKE cluster to create.

• YOUR_BUCKET_NAME: the name of the regional Cloud Storage bucket to create.

• YOUR_HF_TOKEN: the Hugging Face access token that you created in the previous section.

• YOUR_NETWORK_NAME: the network that the GKE cluster uses. Specify one of the following values:

  • If you created a custom network, then specify the name of your network.

  • Otherwise, specify default.

• YOUR_SUBNETWORK_NAME: the subnetwork that the GKE cluster uses. Specify one of the following values:

  • If you created a custom subnetwork, then specify the name of your subnetwork. You can only specify a subnetwork that exists in the same region as the reservation.

  • Otherwise, specify default.

Create a GKE cluster in Autopilot mode

To create a GKE cluster in Autopilot mode, run the following command:

gcloud container clusters create-auto "$CLUSTER_NAME" \
    --project="$PROJECT_ID" \
    --region="$REGION" \
    --release-channel=rapid \
    --network="$NETWORK" \
    --subnetwork="$SUBNETWORK"

Create a Cloud Storage bucket

To create a regional Cloud Storage bucket to store your model, run the following command:

gcloud storage buckets create gs://$GCS_BUCKET_NAME \
    --project=$PROJECT_ID \
    --location=$REGION

Create a Kubernetes secret for Hugging Face credentials

To create a Kubernetes secret for Hugging Face credentials, follow these steps:

1. Configure kubectl to communicate with your GKE cluster:

   gcloud container clusters get-credentials "$CLUSTER_NAME" \
       --location="$REGION" \
       --project="$PROJECT_ID"

2. Create a Kubernetes secret to store your Hugging Face token:

   kubectl create secret generic hf-secret \
       --from-literal=hf_token="${HUGGING_FACE_TOKEN}" \
       --dry-run=client -o yaml | kubectl apply -f -

Configure Workload Identity Federation for Cloud Storage

To allow GKE to securely access the Cloud Storage bucket, set up GKE Workload Identity Federation:

gcloud iam service-accounts create qwen-gcs-sa \
    --project=$PROJECT_ID

gcloud storage buckets add-iam-policy-binding gs://$GCS_BUCKET_NAME \
    --member="serviceAccount:qwen-gcs-sa@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/storage.objectAdmin"

kubectl create serviceaccount qwen-ksa \
    --namespace=default

gcloud iam service-accounts add-iam-policy-binding qwen-gcs-sa@$PROJECT_ID.iam.gserviceaccount.com \
    --project=$PROJECT_ID \
    --role=roles/iam.workloadIdentityUser \
    --member="serviceAccount:$PROJECT_ID.svc.id.goog[default/qwen-ksa]"

kubectl annotate serviceaccount qwen-ksa \
    --namespace=default \
    iam.gke.io/gcp-service-account="qwen-gcs-sa@$PROJECT_ID.iam.gserviceaccount.com"

Populate the Cloud Storage bucket with the Qwen3 model weights

To populate your Cloud Storage bucket with the Qwen3 model weights, run a Kubernetes Job that uses the Cloud Storage FUSE CSI driver to mount your bucket as a volume. The job downloads the model from Hugging Face if it does not already exist in the bucket.

1. Create a file named qwen3-model-loader.yaml with the following content:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: qwen3-model-loader
   spec:
     template:
       metadata:
         annotations:
           gke-gcsfuse/volumes: "true"
           gke-gcsfuse/cpu-limit: "0"
           gke-gcsfuse/memory-limit: "0"
           gke-gcsfuse/ephemeral-storage-limit: "0"
       spec:
         serviceAccountName: qwen-ksa
         restartPolicy: OnFailure
         containers:
         - name: downloader
           image: python:3.11-slim
           resources:
             requests:
               cpu: "4"
               memory: "16Gi"
               ephemeral-storage: "100Gi"
             limits:
               cpu: "4"
               memory: "16Gi"
               ephemeral-storage: "100Gi"
           command: ["/bin/sh", "-c"]
           args:
           - |
             pip install huggingface_hub
             python3 -c '
             import os
             from huggingface_hub import snapshot_download
             model_id = "Qwen/Qwen3-235B-A22B-Instruct-2507"
             local_dir = "/data/Qwen/Qwen3-235B-A22B-Instruct-2507"
             config_path = os.path.join(local_dir, "config.json")
             if os.path.exists(config_path):
                 print(f"Model already exists at {local_dir}. Skipping download.")
             else:
                 print(f"Downloading model {model_id} to {local_dir}...")
                 snapshot_download(
                     repo_id=model_id,
                     local_dir=local_dir,
                     local_dir_use_symlinks=False,
                     token=os.environ.get("HUGGING_FACE_HUB_TOKEN")
                 )
                 print("Download completed successfully!")
             '
           env:
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_token
           volumeMounts:
           - name: gcs-fuse-volume
             mountPath: /data
         volumes:
         - name: gcs-fuse-volume
           csi:
             driver: gcsfuse.csi.storage.gke.io
             volumeAttributes:
               bucketName: $GCS_BUCKET_NAME
               mountOptions: "implicit-dirs"

2. Apply the qwen3-model-loader.yaml manifest to initialize the download job:

   envsubst < qwen3-model-loader.yaml | kubectl apply -f -

3. To verify that the job is running, stream the download job logs:

   kubectl logs -f job/qwen3-model-loader -c downloader
   

4. Wait for the model download job to complete:

   kubectl wait \
       --for=condition=Complete \
       --timeout=1800s job/qwen3-model-loader

5. To delete the job, run the following command:

   kubectl delete job qwen3-model-loader --ignore-not-found

Deploy a vLLM container to your GKE cluster

To deploy the vLLM container to serve the Qwen3 model by using Kubernetes Deployments, do the following:

1. Create a qwen3-235b-deploy.yaml file with your chosen vLLM deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-qwen3-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: qwen3-server
     template:
       metadata:
         labels:
           app: qwen3-server
           ai.gke.io/model: Qwen3-235B-A22B-Instruct-2507
           ai.gke.io/inference-server: vllm
           examples.ai.gke.io/source: user-guide
       spec:
         serviceAccountName: qwen-ksa
         containers:
         - name: qwen-inference-server
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250801_0916_RC01
           resources:
             requests:
               cpu: "10"
               memory: "1000Gi"
               ephemeral-storage: "500Gi"
               nvidia.com/gpu: "8"
             limits:
               cpu: "10"
               memory: "1000Gi"
               ephemeral-storage: "500Gi"
               nvidia.com/gpu: "8"
           command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
           args:
           - --model=$(MODEL_ID)
           - --load-format=runai_streamer
           - --model-loader-extra-config={"distributed":true}
           - --tensor-parallel-size=8
           - --host=0.0.0.0
           - --port=8000
           - --max-model-len=8192
           - --max-num-seqs=4
           - --dtype=bfloat16
           env:
           - name: MODEL_ID
             value: "gs://$GCS_BUCKET_NAME/Qwen/Qwen3-235B-A22B-Instruct-2507"
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom:
               secretKeyRef:
                 name: hf-secret
                 key: hf_token
           volumeMounts:
           - mountPath: /dev/shm
             name: dshm
           livenessProbe:
             httpGet:
               path: /health
               port: 8000
             initialDelaySeconds: 200
             periodSeconds: 10
           readinessProbe:
             httpGet:
               path: /health
               port: 8000
             initialDelaySeconds: 200
             periodSeconds: 5
         volumes:
         - name: dshm
           emptyDir:
             medium: Memory
         nodeSelector:
           cloud.google.com/gke-accelerator: nvidia-b200
           cloud.google.com/reservation-name: $RESERVATION_NAME
           cloud.google.com/reservation-affinity: "specific"
           cloud.google.com/gke-gpu-driver-version: latest
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: qwen3-service
   spec:
     selector:
       app: qwen3-server
     type: ClusterIP
     ports:
       - protocol: TCP
         port: 8000
         targetPort: 8000
   ---
   apiVersion: monitoring.googleapis.com/v1
   kind: PodMonitoring
   metadata:
     name: vllm-qwen3-monitoring
   spec:
     selector:
       matchLabels:
         app: qwen3-server
     endpoints:
     - port: 8000
       path: /metrics
       interval: 30s

2. Apply the qwen3-235b-deploy.yaml file to your GKE cluster:

   envsubst < qwen3-235b-deploy.yaml | kubectl apply -f -

   Because the container uses Run:ai Model Streamer to stream model weights directly from Cloud Storage, the startup time is significantly accelerated.

3. To see the completion status, run the following command:

   kubectl wait \
       --for=condition=Available \
       --timeout=1800s deployment/vllm-qwen3-deployment

   The --timeout=600s flag allows the command to monitor the deployment for up to 10 minutes.

Interact with Qwen3 by using curl

To verify the Qwen3 model that you deployed, do the following:

1. Set up port forwarding to Qwen3:

   kubectl port-forward service/qwen3-service 8000:8000

2. Open a new terminal window. You can then chat with your model by using curl:

   curl http://127.0.0.1:8000/v1/chat/completions \
   -X POST \
   -H "Content-Type: application/json" \
   -d '{
     "model": "Qwen/Qwen3-235B-A22B-Instruct-2507",
     "messages": [
       {
         "role": "user",
         "content": "Describe a GPU in one short sentence?"
       }
     ],
     "stream": false
   }' | jq .

3. The output is similar to the following:

   {
     "id": "chatcmpl-a926ddf7ef2745ca832bda096e867764",
     "object": "chat.completion",
     "created": 1755023619,
     "model": "Qwen/Qwen3-235B-A22B-Instruct-2507",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "A GPU is a specialized electronic circuit designed to rapidly process and render graphics and perform parallel computations.",
           "refusal": null,
           "annotations": null,
           "audio": null,
           "function_call": null,
           "tool_calls": [],
           "reasoning_content": null
         },
         "logprobs": null,
         "finish_reason": "stop",
         "stop_reason": null
       }
     ],
     "service_tier": null,
     "system_fingerprint": null,
     "usage": {
       "prompt_tokens": 16,
       "total_tokens": 36,
       "completion_tokens": 20,
       "prompt_tokens_details": null
     },
     "prompt_logprobs": null,
     "kv_transfer_params": null
   }
   

Observe model performance

If you want to observe your model's performance, then you can use the vLLM dashboard integration in Cloud Monitoring. This dashboard helps you view critical performance metrics for your model like token throughput, network latency, and error rates. For information, see vLLM in the Monitoring documentation.