<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rafa Sanchez on Medium]]></title>
        <description><![CDATA[Stories by Rafa Sanchez on Medium]]></description>
        <link>https://medium.com/@rafaelsf80?source=rss-82391cd06de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XTIW8vweHiBi_9btt7usxA.jpeg</url>
            <title>Stories by Rafa Sanchez on Medium</title>
            <link>https://medium.com/@rafaelsf80?source=rss-82391cd06de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:58:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rafaelsf80/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A review of Arxiv publications on Gemma family models]]></title>
            <link>https://medium.com/google-cloud/a-review-of-arxiv-publications-on-gemma-family-models-ca4baf9ab2ae?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca4baf9ab2ae</guid>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Mon, 09 Dec 2024 00:05:12 GMT</pubDate>
            <atom:updated>2024-12-09T00:05:12.587Z</atom:updated>
            <content:encoded><![CDATA[<p>Google has made a significant contribution in AI to the open community. Libraries like <a href="http://www.tensorflow.org/">TensorFlow</a>, <a href="https://github.com/jax-ml/jax">Jax</a>, <a href="https://github.com/AI-Hypercomputer/maxtext">MaxText</a> or <a href="https://github.com/AI-Hypercomputer/maxdiffusion">MaxDiffusion</a> and models like <a href="https://arxiv.org/abs/2010.11934">mT5</a>, <a href="https://arxiv.org/abs/2010.11934">BERT</a> amongst others have been openly released.</p><p><strong>Gemma</strong> is not just a model, it’s a family of state-of-the-art models and tools built from the same research and technology used to create Google’s Gemini models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xQyXKlWq4TD332_9YsIXhA.jpeg" /></figure><p>This post is a summary of <strong>all the Arxiv papers</strong> describing the variety of innovations introduced in Gemma:</p><ul><li><a href="https://arxiv.org/abs/2403.08295">Gemma 1</a>: first LLM model family announced in February 2024, a decoder-only transformer with 18/28 layers (2B/7B models respectively), GeLU activation functions and a vocab size of 256k tokens. Text-only, English-only and with an input window of 8K tokens.</li><li><a href="https://arxiv.org/abs/2404.07839">RecurrentGemma</a>: variant of Gemma 1 that includes the innovative Griffin architecture, which essentially replaces the attention layer of the transformer by a new Linear Recurrent layer, with the objective of reduce the quadratical complexitity of a standard transformer.</li><li><a href="https://arxiv.org/abs/2402.19427">Griffin</a>: detailed description of the Griffin architecture, which is the basis of the RecurrentGemma model. Griffin combines mixes gated linear recurrences with local attention blocks.</li><li><a href="https://arxiv.org/abs/2406.11409">CodeGemma 1</a>: variant of Gemma 1 with an extra fine-tuning for coding tasks.</li><li><a href="https://arxiv.org/abs/2407.07726">PaliGemma 1</a>: 3B Vision-Language Model (VLM) to generate text from images, which implements SigLP vision encoder and the Gemma 1–2B language model.</li><li><a href="https://arxiv.org/abs/2303.15343">SigLIP</a>: is one of the key technologies behind PaliGemma. Unlike standard contrastive learning with softmax normalization, this new technique uses the sigmoid loss only on image-text pairs and does not require a global view of the pairwise similarities for normalization.</li><li><a href="https://arxiv.org/abs/2408.00118">Gemma 2</a>: new model family announced in May 2024 (just four months after Gemma 1), with three sizes 2B, 9B and 27B. Main architectural differences include interleaving local-global attention layers and group-query attention. Another interesting topic is that the smaller 2B and 9B models are trained with knowledge distillation.</li><li><a href="https://arxiv.org/abs/2407.21772">ShieldGemma</a>: content moderation models built on Gemma 2.</li><li><a href="https://arxiv.org/abs/2408.05147">Gemma Scope</a>: interpretability is a challenge in LLM, due to architecture and computational requirements. Gemma Scope is a collection of hundreds of open sparse autoencoders (SAEs), that can help to understand the inner workings of for Gemma 2–9B and Gemma 2–2B (only those two models for now).</li><li><a href="https://arxiv.org/abs/2409.13741">DataGemma</a>: fine-tuned Gemma 2 models grounded to the real-world data of Data Commons.</li><li><a href="https://arxiv.org/abs/2412.03555">PaliGemma 2</a>: new model announced in December 2024 updated with Gemma 2–2B, 9B and 27B while keeping the same SigLIP encoder used in PaliGemma 1. Main improvements include long detailed captioning capabilities, as well as advanced vision use cases like music score recognition or chest X-ray report generation.</li></ul><p>Non-Google researchers have also contributed to the <strong>Gemma ecosystem</strong> with papers:</p><ul><li><a href="https://www.arxiv.org/abs/2407.02147">GemmAr</a>: describes InstAr-500k, a new Arabic instruction dataset, and the subsequent finetuning process of Gemma 1–7B on several downstream tasks.</li><li><a href="https://arxiv.org/abs/2407.01449">ColPali</a>: a PaliGemma 1–3B extension which implements document understanding capabilities, especially for images of documents.</li></ul><h3><strong>Deployment in Vertex AI</strong></h3><p>Deploying open models should not be difficult and has never been so easy with a cloud platform like <strong>Vertex AI</strong>, which allows deployment, fine-tuning, evaluation and many other tasks on Gemma.</p><ul><li>For Gemma 1, you can use the free (and no sign-in required) demo playground in Vertex AI Model Garden <a href="https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemma">here</a>.</li><li>For Gemma 2, you can use the free demo playground in Vertex AI Model Garden <a href="https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemma2">here</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DDpLlsqW10XjUwXi" /><figcaption>Fig. 1 Demo playground in Vertex AI Model garden</figcaption></figure><p>Additionally, you can also find pre-trained models and code on Hugging Face and Kaggle.</p><h3>References</h3><p>[1] Blog post <a href="https://blog.google/technology/developers/gemma-open-models/">Gemma 1 announcement</a></p><p>[2] Blog post <a href="https://developers.googleblog.com/en/gemma-family-expands-with-models-tailored-for-developers-and-researchers/">New Gemma 1 variants</a></p><p>[3] Blog post <a href="https://developers.googleblog.com/en/gemma-family-and-toolkit-expansion-io-2024/">PaliGemma 1, Gemma 2, and an Upgraded Responsible AI Toolkit</a></p><p>[4] Blog post <a href="https://developers.googleblog.com/en/gemma-explained-overview-gemma-model-family-architectures/">Gemma 1 explained: Gemma models family architectures</a></p><p>[5] Blog post <a href="https://developers.googleblog.com/en/gemma-explained-recurrentgemma-architecture/">Gemma 1 explained: RecurrentGemma Architectures</a></p><p>[6] Blog post <a href="https://developers.googleblog.com/en/gemma-explained-paligemma-architecture/">Gemma 1 explained: PaliGemma Architectures</a></p><p>[7] Blog post <a href="https://developers.googleblog.com/en/gemma-explained-new-in-gemma-2/">Gemma 1 explained: What&#39;s new in Gemma 2</a></p><p>[8] Blog post <a href="https://developers.googleblog.com/en/advancing-multilingual-ai-with-gemma-2-and-a-150k-challenge/">Advancing Multilingual AI with Gemma 2 and a $150K Challenge</a></p><p>[9] Blog post <a href="https://developers.googleblog.com/en/smaller-safer-more-transparent-advancing-responsible-ai-with-gemma/">ShieldGemma and Gemma Scope</a></p><p>[10] Blog post <a href="https://blog.google/technology/ai/google-datagemma-ai-llm/">DataGemma</a></p><p>[11] Blog post <a href="https://developers.googleblog.com/en/introducing-paligemma-2-powerful-vision-language-models-simple-fine-tuning/">PaliGemma 2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca4baf9ab2ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/a-review-of-arxiv-publications-on-gemma-family-models-ca4baf9ab2ae">A review of Arxiv publications on Gemma family models</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Remote to Vertex AI Workbench Instances over an IAP tunnel with VS Code]]></title>
            <link>https://medium.com/google-cloud/remote-to-vertex-ai-workbench-instances-over-an-iap-tunnel-with-vs-code-5c052f2cc7fa?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c052f2cc7fa</guid>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[gcp-app-dev]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Thu, 26 Oct 2023 20:18:02 GMT</pubDate>
            <atom:updated>2023-11-10T18:24:59.135Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post you will learn how to use your VS Code client as IDE in your local development machine while using <a href="https://cloud.google.com/vertex-ai/docs/workbench/instances/introduction">Vertex AI Workbench Instances</a> as remote, all of that within a protected environment with an IAP tunnel.</p><p><a href="https://cloud.google.com/vertex-ai/docs/workbench/instances/introduction">Vertex AI Workbench Instances</a> is the new enterprise-grade Jupyter notebook environment for data scientists. It’s currently in General availability (GA) in Google Cloud Platform. Main features include the following:</p><ul><li>Large number of <strong>hardware configurations</strong>, including <a href="https://cloud.google.com/vertex-ai/docs/workbench/instances/create#create">multiple GPU types</a>.</li><li><strong>BigQuery Browser/Query Editor:</strong> browse BigQuery Tables and author SQL from within the JupyterLab.</li><li><strong>Dataproc Support:</strong> run computations on Dataproc clusters and Dataproc Serverless Sessions.</li><li><strong>Notebook Executor: </strong>schedule the execution of notebooks on Training service.</li><li><strong>Idle Shutdown:</strong> saves money by automatically suspending idle notebooks.</li><li><strong>Google Cloud Theme: </strong>a new Google Cloud theme bringing a modern UI to JupyterLab.</li><li><strong>Org Policies:</strong> admins can set defaults and turn off features of Notebooks.</li><li><strong>Multi-kernel:</strong> users can leverage multiple research environments from a single instance and UI.</li><li>Support for <strong>Sec4 horizontals</strong> (CMEK, AXT, DRZ, VPC-SC).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*TLbsHXsv_xwy1aaeh8A-Yg.png" /><figcaption>Fig. 1 Jupyter logo (source: jupyter.org)</figcaption></figure><p>The steps below only apply to Vertex AI Workbench Instances, not to other type of notebook-related services, like managed notebooks.</p><p>You actually do not need an OAuth token, but private/public keys. You will also use Identity-Aware Proxy (IAP) to protect SSH access to the notebook VM via <a href="https://cloud.google.com/iap/docs/tcp-forwarding-overview/">TCP forwarding</a>. Using IAP, the VM instances containing the notebook don’t even need a public IP address.</p><p>These are the five steps:</p><p>Step 1:</p><p>Create the Vertex AI Workbench Instance, in my case with instance name my-vertex-iap-instance. Make sure you disable the external IP address. You can optionally configure GPUs in this step.</p><p>Step 2:</p><p>Once created, set IAP in your Google Cloud project. Click on IAP in the console and select <strong>SSH and TCP resources</strong>. Select the VM corresponding to your Vertex AI Workbench Instance. Follow the instructions <a href="https://cloud.google.com/iap/docs/using-tcp-forwarding">here</a> and make sure you set the proper permissions.</p><p>Step 3:</p><p>In your local dev machine, make sure gcloud config list returns the right project_id and credentials. In this case we can not automatically populate the config ssh file with gcloud compute config-ssh because we are using an internal IP address for our instance. We need to get the proper SSH configuration with this command:</p><pre>gcloud compute ssh my-vertex-iap-instance --dry-run<br><br># Output for Mac OSX:<br>/usr/local/bin/ssh -t -i /Users/rafaelsanchez/.ssh/google_compute_engine -o CheckHostIP=no -o HashKnownHosts=no -o HostKeyAlias=compute.6850135882903047257 -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/Users/rafaelsanchez/.ssh/google_compute_known_hosts -o &quot;ProxyCommand /usr/local/bin/python3 -S /Users/rafaelsanchez/Library/Application\ Support/cloud-code/installer/google-cloud-sdk/lib/gcloud.py compute start-iap-tunnel my-vertex-iap-instance %p --listen-on-stdin --project=argolis-rafaelsanchez-ml-dev --zone=europe-west4-a --verbosity=warning&quot; -o ProxyUseFdpass=no admin_rafaelsanchez_altostrat_co@compute.6850135882903047257</pre><p>Note the output of the previous command. We need to parse the fields into entries for the local SSH config file local (~/.ssh/config). We can do that automatically with the help of VS Code by clicking <em>CMD+P</em> (Mac OSX) or <em>CTRL+P</em> (Windows), select Remote-SSH: Add new SSH Host and paste the previous long SSH command (the one starting with /usr/local/bin/ssh -t -i [...]):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MMKKpQPeRt6JMtHOP0iGSQ.png" /><figcaption>Fig. 2 Enter SSH command</figcaption></figure><p>Then select the config file (in my case /Users/rafaelsanchez/.ssh/config):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5N1MSe4Yz864sWiVWfeZCQ.png" /><figcaption>Fig. 3 Selecting the SSH config file</figcaption></figure><p>A Host added message window should appear:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGX4k6OphBBYn5R3CmFHuw.png" /><figcaption>Fig. 4 Host added message</figcaption></figure><p>If you click <strong>Open Config</strong>, you can see the preliminary new entry that has been automatically added to ~/.ssh/config. You will need to make some manual corrections to that entry in the next steps before connecting to the instance.</p><pre>Host /usr/local/bin/ssh<br>    HostName /usr/local/bin/ssh<br>    IdentityFile /Users/rafaelsanchez/.ssh/google_compute_engine<br>    CheckHostIP no<br>    HashKnownHosts no<br>    HostKeyAlias compute.6850135882903047257<br>    IdentitiesOnly yes<br>    StrictHostKeyChecking yes<br>    UserKnownHostsFile /Users/rafaelsanchez/.ssh/google_compute_known_hosts<br>    ProxyCommand /usr/local/bin/python3 -S /Users/rafaelsanchez/Library/Application\ Support/cloud-code/installer/google-cloud-sdk/lib/gcloud.py compute start-iap-tunnel my-vertex-iap-instance %p --listen-on-stdin --project argolis-rafaelsanchez-ml-dev --zone=europe-west4-a --verbosity=warning<br>    ProxyUseFdpass no</pre><p>Step 4:</p><p>In order to use the jupyter user of your notebook, you must create a public/private key and then upload public key to the remote instance. To create public/private key, you can do it in your local dev machine:</p><pre>ssh-keygen -t rsa -f ~/.ssh/gcp-jupyter -C jupyter<br>chmod 400 ~/.ssh/gcp-jupyter</pre><p>Place the private key to your local dev machine in ~/.ssh/gcp-jupyter. Upload manually the public key to Vertex AI Workbench Instances in ~/.ssh/gcp-jupyter.pub, <strong>renaming the file</strong> as ~/.ssh/authorized_keys.</p><p>Finally, you need to manually modify the jupyter user on your local ~/.ssh/config, under User and IdentityFile fields. You need also to modify the Host and HostName fields. The final entry will look like this:</p><pre>Host my-vertex-iap-instance<br>    HostName compute.6850135882903047257<br>    User jupyter<br>    IdentityFile /Users/rafaelsanchez/.ssh/gcp-jupyter<br>    CheckHostIP no<br>    HashKnownHosts no<br>    HostKeyAlias compute.6850135882903047257<br>    IdentitiesOnly yes<br>    StrictHostKeyChecking yes<br>    UserKnownHostsFile /Users/rafaelsanchez/.ssh/google_compute_known_hosts<br>    ProxyCommand /usr/local/bin/python3 -S /Users/rafaelsanchez/Library/Application\ Support/cloud-code/installer/google-cloud-sdk/lib/gcloud.py compute start-iap-tunnel my-vertex-iap-instance %p --listen-on-stdin --project argolis-rafaelsanchez-ml-dev --zone=europe-west4-a --verbosity=warning<br>    ProxyUseFdpass no</pre><p>Step 5:</p><p>You are now ready to connect from VS Code via SSH. In VS Code, <em>CMD+P</em> (Mac OSX) or <em>CTRL+P</em> (Windows) and write Remote-SSH: Connect to Host and select the right notebook VM (in my case my-vertex-iap-instance):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4Yx3gWpEWI-8L0gfKAsBA.png" /><figcaption>Fig. 5 Connecting to host</figcaption></figure><p>You should see <strong>SSH: my-vertex-iap-instance</strong> at the bottom left following a successful connection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXMMpcyN4KDpfrUi6tWGxw.png" /><figcaption>Fig. 6 Connection successful with IAP</figcaption></figure><p>Some notes:</p><ul><li>Make sure you disable external IP address on Vertex AI Workbench Instances.</li><li>Make sure you rename the public key as .ssh/authorized_keys in the remote instance.</li><li>Make sure the ProxyCommand is properly written, for example, this gives a File not found errordue to the backslash before Support: `ProxyCommand /usr/local/bin/python3 -S /Users/rafaelsanchez/Library/Application\ Support/cloud-code/installer/google-cloud-sdk/lib/gcloud.py compute start-iap-tunnel my-vertex-iap-instance %p — listen-on-stdin — project argolis-rafaelsanchez-ml-dev — zone=europe-west4-a — verbosity=warning`</li></ul><h3>References</h3><p>[1] VS Code documentation: <a href="https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_connect-to-a-remote-jupyter-server">Connect to a remote jupyter server</a><br>[2] Medium article: <a href="https://medium.com/@albert.brand/remote-to-a-vm-over-an-iap-tunnel-with-vscode-f9fb54676153">Remote to a VM over an IAP tunnel with VSCode</a><br>[3] Identity-Aware Proxy (IAP) <a href="https://cloud.google.com/iap/docs/concepts-overview">overview</a><br>[4] Vertex AI Workbench Instances <a href="https://cloud.google.com/vertex-ai/docs/workbench/instances/introduction">overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c052f2cc7fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/remote-to-vertex-ai-workbench-instances-over-an-iap-tunnel-with-vs-code-5c052f2cc7fa">Remote to Vertex AI Workbench Instances over an IAP tunnel with VS Code</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI — Deploy and inference of Llama 2 in Vertex AI Prediction]]></title>
            <link>https://medium.com/google-cloud/generative-ai-deploy-and-inference-of-llama-2-in-vertex-ai-prediction-97fae2472dd5?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/97fae2472dd5</guid>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Sun, 27 Aug 2023 16:33:16 GMT</pubDate>
            <atom:updated>2023-10-09T16:58:56.088Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generative AI — Deploy and inference of Llama 2 in Vertex AI Prediction</h3><p>This post shows how to <strong>deploy a Llama 2 chat model</strong> (7B parameters) in Vertex AI Prediction with a T4 GPU. The model will be downloaded and embedded in a custom prediction image, using an <a href="https://www.uvicorn.org/">Uvicorn</a> server. You will use a n1-standard-4 machine type with 1xT4 NVidia GPU in <strong>Vertex AI Prediction</strong>.</p><p>A demo based on <a href="https://streamlit.io/">Streamlit</a> and deployed in <a href="https://cloud.google.com/run">Cloud Run</a> is also provided to easily make requests into the deployed model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RaQuKcuXwKgA4qWn--bDbQ.png" /></figure><h3>The model: Llama 2</h3><p><a href="https://ai.meta.com/llama/">Llama 2</a> is the next-generation of Meta Large Language Model, released with a <a href="https://github.com/facebookresearch/llama/blob/main/LICENSE">free license for commercial and educational use</a>. Note Clause 2 related the limitation of 700 million monthly active users, that requires a Meta specific license.</p><p>Llama 2 is a transformer-based auto-regressive language model, with a number of model sizes. Main features of this model are:</p><ul><li>Trained on a new mix of publicly available data, in total around 2T tokens.</li><li>Doubled the context length of the model (4K) compared to Llama 1 (2K).</li><li>Grouped-query attention.</li><li>Available sizes: 7B, 13B, 34B (pretrained only, not chat), 70B parameters.</li></ul><p>Model card <a href="https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md">here</a> and paper <a href="https://arxiv.org/abs/2307.09288">here</a>.</p><p><a href="https://github.com/ggerganov/ggml">GGML</a> is a tensor library for deep learning writen in C, without extra dependencies of other known deep learning libraries like Torch, Transformers, Accelerate. GGML only needs CUDA/C++ for GPU execution.</p><p>GGML implements some features to train and optimize LLM models on commodity hardware.</p><p><a href="https://github.com/marella/ctransformers">CTransformers</a> is a python bind for GGML.</p><p>In this post we will use the GGML weights converted by <a href="https://huggingface.co/TheBloke">TheBloke</a> and stored in Hugging Face. The model used in this example is the 2-bit GGML model, that uses the <a href="https://github.com/ggerganov/llama.cpp/pull/1684">new k-quant method</a> in GGML.</p><p>You must <a href="https://huggingface.co/TheBloke/Dolphin-Llama2-7B-GGML/tree/main">download the model</a> and store it in predict/llama2-7b-chat-ggml directory, with a similar content like this. Note no need to use a handler.py since you will not use TorchServe:</p><pre>config.json<br>dolphin-llama2-7b.ggmlv3.q2_K.bin</pre><h3>Build Custom Prediction Container image</h3><p>A Custom Container image for predictions is required. A Custom Container image in Vertex AI <a href="https://cloud.google.com/ai-platform-unified/docs/predictions/custom-container-requirements#image">requires that the container must run an HTTP server</a>. Specifically, the container must listen and respond to liveness checks, health checks, and prediction requests.</p><p>You will use a <a href="https://www.uvicorn.org/">Uvicorn</a> server. You must build and push the container image to <a href="https://cloud.google.com/artifact-registry">Artifact Registry</a>:</p><pre>gcloud auth configure-docker europe-west4-docker.pkg.dev<br>gcloud builds submit --tag europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/llama2-7b-chat-q2 --machine-type=e2-highcpu-8 --timeout=&quot;2h&quot; --disk-size=300</pre><p>This build process should take <strong>less than 20 minutes</strong> with a e2-highcpu-8.</p><h3>Deploy the model to Vertex AI Prediction</h3><p>Upload and deploy the image to Vertex AI Prediction using the provided script: python3 upload_custom.py.</p><p>The upload and deploy process <strong>may take up to 45 min</strong>. Note the parameter deploy_request_timeout to avoid a 504 Deadline Exceeded error during the deployment:</p><pre>DEPLOY_IMAGE = &#39;europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/llama2-7b-chat-q2&#39; <br>HEALTH_ROUTE = &quot;/health&quot;<br>PREDICT_ROUTE = &quot;/predict&quot;<br>SERVING_CONTAINER_PORTS = [7080]</pre><pre>model = aiplatform.Model.upload(<br>    display_name=f&#39;llama2-7B-chat-q2&#39;,    <br>    description=f&#39;llama2-7B-chat-q2 with Uvicorn and FastAPI&#39;,<br>    serving_container_image_uri=DEPLOY_IMAGE,<br>    serving_container_predict_route=PREDICT_ROUTE,<br>    serving_container_health_route=HEALTH_ROUTE,<br>    serving_container_ports=SERVING_CONTAINER_PORTS,<br>)<br>print(model.resource_name)</pre><pre># Retrieve a Model on Vertex<br>model = aiplatform.Model(model.resource_name)</pre><pre># Deploy model<br>endpoint = model.deploy(<br>    machine_type=&#39;n1-standard-4&#39;, <br>    accelerator_type= &quot;NVIDIA_TESLA_T4&quot;,<br>    accelerator_count = 1,<br>    traffic_split={&quot;0&quot;: 100}, <br>    min_replica_count=1,<br>    max_replica_count=1,<br>    traffic_percentage=100,<br>    deploy_request_timeout=1200,<br>    sync=True,<br>)<br>endpoint.wait()</pre><p>The <a href="https://cloud.google.com/vertex-ai/pricing#prediction-prices">cost of a Vertex Prediction endpoint</a> (24x7) is splitted between <strong>vCPU cost</strong> (measured in vCPU hours), <strong>RAM cost</strong> (measured in GB hours) and <strong>GPU cost</strong> (measured in hours). In this case, we will use a n1-standard-4 with 1xT4 GPU in europe-west4 and the estimated cost is (0.4370 + 0.02761*4 + 0.0037*16)*24*30 = 436.78 USD per month.</p><h3>Streamlit demo UI</h3><p>You are now ready to predict on the deployed model. You can use the REST API or the python SDK, but in this case you will build a simple demo UI using <a href="https://streamlit.io/">Streamlit</a>. There are many other similar UI libraries, like Gradio, but this time let’s use Streamlit.</p><p>The Streamlit app is built on <a href="https://cloud.google.com/run">Cloud Run</a>. You need to <a href="https://docs.streamlit.io/knowledge-base/tutorials/deploy/docker">build the docker first</a>, upload it to Artifact registry and then deploy it in Cloud Run:</p><pre>gcloud builds submit --tag europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/llama2-7b-chat-streamlit <br>gcloud run deploy llama2-7b-chat-streamlit --port 8501 --image europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/llama2-7b-chat-streamlit --allow-unauthenticated --region=europe-west4 --platform=managed</pre><blockquote><em>NOTE: in the last two sections you have created two dockers, one </em><strong><em>to host the Llama 2 model</em></strong><em> (a custom container image which is then deployed in Vertex AI) and the other one (the one in this section) </em><strong><em>to host the Streamlit app</em></strong><em> that will call the model.</em></blockquote><p>The Streamlit app is now deployed in Cloud Run. You can test the provided examples ot try yours. Please, note the purpose of this post is not to evaluate the performance of the Llama 2–7B chat model, but to show how to deploy a model like this in Vertex AI and Cloud Run. You may get better results with higher quantized versions or bigger models like Llama 2–13B.</p><h3>Model garden</h3><p><a href="https://cloud.google.com/model-garden">Model garden on Vertex AI</a> is a single place to search, discover, and interact with a wide variety of models from Google and others, including open source models.</p><p>At the time of writing this post, Llama 2 is still not available in Model garden. Only OpenLlama, a reproduction of Llama 1, is available in Model garden, with code samples for deployment and tuning in Vertex AI.</p><p>UPDATE August 30th, 2023: as <a href="https://cloud.google.com/blog/topics/google-cloud-next/next-23-recap-day-1">announced in Cloud Next23 event</a>, Llama 2 is now available in Model garden.</p><h3>Conclusions</h3><p>This post shows how to make deployment of the latest Llama 2 model in Vertex AI. It uses Vertex AI prediction with one single GPU and exposes the model through a Streamlit app.</p><p>You can find the repo with all the code in <a href="https://github.com/rafaelsf80/genai-llama2-infer-vertex">this link</a>.</p><p><strong>UPDATE</strong> 09-october-2023: <a href="https://github.com/rafaelsf80/genai-llama2-70b-infer-vertex">here</a> is a Github repo to deploy <strong>Llama 2 Chat 70B</strong> in Vertex AI Prediction with 2xL4 GPU.</p><h3>References</h3><p>[1] Research paper: <a href="https://arxiv.org/abs/2307.09288">Llama 2</a><br>[2] Original <a href="https://github.com/facebookresearch/llama#download">Llama 2 Checkpoints</a><br>[3] Medium post: <a href="https://medium.com/streamlit/how-to-build-an-llm-powered-chatbot-with-streamlit-a1bf0b2701e8">How to build an LLM-powered chatbot with Streamlit</a><br>[4] Medium post: <a href="https://vilsonrodrigues.medium.com/run-llama-2-models-in-a-colab-instance-using-ggml-and-ctransformers-41c1d6f0e6ad">LlaMa 2 models in a Colab instance using GGML and ctransformers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97fae2472dd5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/generative-ai-deploy-and-inference-of-llama-2-in-vertex-ai-prediction-97fae2472dd5">Generative AI — Deploy and inference of Llama 2 in Vertex AI Prediction</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI — Q&A with semantic answering on large scanned documents with Vertex AI, Chroma…]]></title>
            <link>https://medium.com/google-cloud/generative-ai-q-a-with-semantic-answering-on-large-scanned-documents-with-vertex-ai-chroma-7f4806a3cb71?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f4806a3cb71</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gene]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Wed, 21 Jun 2023 11:13:21 GMT</pubDate>
            <atom:updated>2023-06-22T08:20:29.451Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generative AI — Q&amp;A with semantic answering on large scanned documents with Vertex AI, Chroma, LangChain and Document AI OCR</h3><p>The post shows how to make<strong> semantic question and answering</strong> over large scanned documents and pdfs, like mortgages, contracts or large finantial documents. The documents can be pdf or images in any of these <a href="https://cloud.google.com/document-ai/docs/file-types">supported file formats</a>.</p><p>The <a href="https://github.com/rafaelsf80/genai-vertex-documents-asynchronous">code repo</a> includes a <a href="https://gradio.app/">Gradio app</a> to ask large docs by showing the combined capabilities of Vertex LLM models, a vector store like <a href="https://github.com/chroma-core/chroma">Chroma</a>, <a href="https://github.com/hwchase17/langchain">LangChain</a> and <a href="https://cloud.google.com/document-ai/docs/samples/documentai-process-ocr-document#documentai_process_ocr_document-python">Google Cloud Document AI OCR</a>. So, basically it implements a semantic search on large and unstructured documents, by converting them to text first using the OCR engine of Document AI, and them implementing a retrieval augmented process with an LLM.</p><p>The pdf sample used is the public <em>2022 BBVA annual report</em> (in English), a<strong> 564-page pdf</strong> that can be downloaded from <a href="https://shareholdersandinvestors.bbva.com/wp-content/uploads/2023/03/Annual-Report-BBVA_2022_ENG.pdf">here</a>. You can replace this doc with any other, like mortgages or any other large documents in the Finance and Insurance industry, even if they contain images or scanned information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oCX1Pb-rojTGQ7Kqs2MiiQ.png" /><figcaption>Fig.1 Gradio app with semantic search on large scanned docs</figcaption></figure><h3>Document AI OCR (batch processing)</h3><p>The first step is to download the public document and uploading into GCS, since this is required for OCR batch processing. Refer to constant gcs_input_uri in local/ocr_batch.py in the code repo and upload the pdf file in GCS.</p><p>Note also you need to create a Document AI OCR processor and replace the constants project_id, region and processor_id.</p><p>Why using Document AI OCR and not open-source tools like PyPDF to manage the document ?</p><p>The reason is to be able also to process documents containing text coming from scanning or photos, since PyPDF and other similar tools will not make OCR even if the extension is .pdf.</p><p>Therefore, to extract the text from a photo or a scanned documents, you must use OCR tools like Document AI OCR. <a href="https://cloud.google.com/document-ai/docs">Google Cloud Document AI</a> is a document understanding solution that takes unstructured data (e.g. documents, emails, invoices, forms, etc.) and makes the data easier to understand, analyze, and consume. One of the pre-trained models performs <strong>Optical Character Recognition (OCR)</strong>, which is the one we will use in this post. According to <a href="http://dx.doi.org/10.1007/s42001-021-00149-1">this paper</a>, Document AI OCR from Google outperforms other OCR solutions available.</p><p>Document AI OCR parser supports <strong>online and batch processing</strong>. Since we are dealing with large documents (more than 15 pages, which is the <a href="https://cloud.google.com/document-ai/quotas#general_processors">current limit for online processing</a>), we will be using batch mode.</p><p>The script local/ocr_batch.py extracts text from a public doc, in this case the English version of the 2002 BBVA annual report (publicly available), a document with 564 pages. Since the extraction by Document AI OCR is in multiple chunks, they must be merged in a single file output_all.txt which is also generated by the same script.</p><pre>cd local<br>python3 ocr_batch.py</pre><h3>Indexing and retrieval with Chroma and LangChain</h3><p>Once you get the OCR text output_all.txt from the first step, you must create the index in <a href="https://github.com/chroma-core/chroma">Chroma</a> (note the store is done persistently), by executing python3 create_index_chroma.py:</p><pre>python3 create_index_chroma.py</pre><p>Once the index is built, you are ready to<strong> launch the retrieval process</strong>. The retrieval process is similar to the one described in <a href="http://arxiv.org/abs/2007.01282">this paper</a>, which proposes a straight-forward approach for combining retrieval systems with large language models for question answering. Essentially, these are the steps:</p><ol><li>The end-user makes a question.</li><li>The retrieval system finds two passages (number of passages is configurable in the code) from the original text.</li><li>The two passages are concatenated with the user question.</li><li>All that concatenated info (user question and two passages) is sent to the PaLM-2 model text-bison@001. The paper names the process of concatenating the question and the two passages as <em>fusion-in-the-decoder</em>, after which the answer is returned to the user.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*cOERMXRpLDZZFddJI_tyww.png" /><figcaption>Fig.2 Fusion-in-the-decoder (Source: <a href="http://arxiv.org/abs/2007.01282">http://arxiv.org/abs/2007.01282</a>)</figcaption></figure><p>To launch the retrieval, run python3 retrieval_chroma.py. This script uses <strong>Retrieval QA chain in LangChain </strong>to recover the blocks closer to the query, which includes a Vertex LLM model (PaLM-2) to generate the output to the user.</p><pre>python3 retrieval_chroma.py</pre><h3>End-to-end application in Gradio</h3><p>The Gradio app will implement the full cycle described above in a single app:</p><ol><li>Load a pdf file</li><li>Perform <strong>batch processing</strong> with Document AI OCR parser, converting to text.</li><li>Create the index with Chroma.</li><li>After 20–30 min (due to batch processing in Document AI OCR), the user can then write the query, and the Retrieval QA chain will recover the passages closer to the query. A Vertex LLM model will write the output to the user.</li></ol><blockquote>NOTE: the full cycle described above can take <strong>up to 30 minutes</strong> before you can query the doc. Main time consumption is due to the Document AI OCR batch processing.</blockquote><p>Refer to <a href="https://medium.com/google-cloud/generative-ai-palm-2-model-deployment-with-cloud-run-54e8a398b24b">this Medium post</a> on how to set the Cloud Run service account with minimum permissions. On top of those minimum permissions, you will need the following extra roles for this app: roles/logging.privateLogViewer, roles/storage.admin (create buckets) and roles/documentai.apiUser.</p><pre># extra roles required on top of minimum permissions<br>PROJECT_ID=&lt;REPLACE_WITH_YOUR_PROJECT_ID&gt;<br><br># add logging.privateLogViewer role<br>gcloud projects add-iam-policy-binding $PROJECT_ID \<br>    --member=&quot;serviceAccount:cloud-run-llm@${PROJECT_ID}.iam.gserviceaccount.com&quot; \<br>    --role=&quot;roles/logging.privateLogViewer&quot;<br><br># add documentai.apiUser role <br>gcloud projects add-iam-policy-binding $PROJECT_ID \<br>    --member=&quot;serviceAccount:cloud-run-llm@${PROJECT_ID}.iam.gserviceaccount.com&quot; \<br>    --role=&quot;roles/documentai.apiUser&quot;<br><br># add storage.admin role<br>gcloud projects add-iam-policy-binding $PROJECT_ID \<br>    --member=&quot;serviceAccount:cloud-run-llm@${PROJECT_ID}.iam.gserviceaccount.com&quot; \<br>    --role=&quot;roles/storage.admin&quot;</pre><p>To build and deploy the <a href="https://gradio.app/">Gradio app</a> in <a href="https://cloud.google.com/run/docs/quickstarts/deploy-container">Cloud Run</a>, execute the following commands. Note authentication is disabled and the service account is the one configured earlier:</p><pre>PROJECT_ID=&lt;REPLACE_WITH_YOUR_PROJECT_ID&gt;<br>REGION=&lt;REPLACE_WITH_YOUR_GCP_REGION_NAME&gt;<br>AR_REPO=&lt;REPLACE_WITH_YOUR_AR_REPO_NAME&gt;<br>SERVICE_NAME=docai-large<br><br>gcloud artifacts repositories create $AR_REPO --location=$REGION --repository-format=Docker<br>gcloud auth configure-docker $REGION-docker.pkg.dev<br>gcloud builds submit --tag $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$SERVICE_NAME<br>gcloud run deploy $SERVICE_NAME --port 7860 --image $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$SERVICE_NAME --service-account=cloud-run-llm@$PROJECT_ID.iam.gserviceaccount.com --allow-unauthenticated --region=europe-west4 --platform=managed  --project=$PROJECT_ID</pre><h3>Conclusions</h3><p>This post shows how to make semantic search on large scanned documents using LLM models like PaLM-2 in Vertex AI, together with open-source tools like Chroma and LangChain.</p><p>The post described how to implement the semantic search step-by-step in your local machine, and also with an end-to-end app deployed in Cloud Run using Gradio.</p><p>You can find the repo with all the code in <a href="https://github.com/rafaelsf80/genai-vertex-documents-asynchronous">this link</a>.</p><h3>References</h3><p>[1] Medium post: <a href="https://medium.com/google-cloud/generative-ai-palm-2-model-deployment-with-cloud-run-54e8a398b24b">Generative AI - PaLM-2 model deployment with Cloud Run</a><br>[2] Paper: <a href="http://arxiv.org/abs/2007.01282">Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering</a><br>[3] Paper: <a href="http://arxiv.org/abs/2005.11401">Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks</a><br>[4] YouTube video: <a href="https://youtu.be/Q1zF9pF6flw">Generative AI on Google Cloud</a><br>[5] Documentation: <a href="https://cloud.google.com/document-ai/docs/samples/documentai-process-ocr-document#documentai_process_ocr_document-python">Google Cloud Document AI OCR</a><br>[6] LangChain documentation: <a href="https://python.langchain.com/docs/modules/model_io/models/llms/integrations/google_vertex_ai_palm">Google Cloud Platform Vertex AI PaLM</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f4806a3cb71" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/generative-ai-q-a-with-semantic-answering-on-large-scanned-documents-with-vertex-ai-chroma-7f4806a3cb71">Generative AI — Q&amp;A with semantic answering on large scanned documents with Vertex AI, Chroma…</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI — PaLM-2 model deployment with Cloud Run]]></title>
            <link>https://medium.com/google-cloud/generative-ai-palm-2-model-deployment-with-cloud-run-54e8a398b24b?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/54e8a398b24b</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Fri, 26 May 2023 17:06:08 GMT</pubDate>
            <atom:updated>2023-05-29T05:42:11.919Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generative AI — PaLM-2 model deployment with Cloud Run</h3><p>This post shows a frontend in <a href="https://gradio.app/">Gradio</a>, deployed in <a href="https://cloud.google.com/run/docs/quickstarts/deploy-container">Cloud Run</a>, that exposes one of the PaLM-2 foundational models text-bison@001.</p><p>text-bison@001 is <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models">one of the foundational models</a> based on PaLM-2 that is available in Vertex AI. This post shows a front-end exposing this model and <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models#parameter_definitions">its main parameters</a> (temperature, output tokens, top-P and top-K) via a Gradio app.</p><p>The model text-bison@001 is <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview#palm-api">fine-tuned</a> for language tasks such as classification, summarization, and entity extraction.</p><p>The frontend is deployed through a Gradio app in <a href="https://cloud.google.com/run">Cloud Run</a>. A screenshot of the app follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sVgzE2oglTjiHGd6gKeRLA.png" /><figcaption>Fig.1 Gradio app exposing PaLM-2 model with its main parameters: temp, output, topP, topK</figcaption></figure><h3>PaLM-2 in Vertex AI</h3><p>text-bison@001 is one of the foundational models available in Vertex AI, based on PaLM-2, and fine-tuned for certain language tasks. Details about PaLM-2 can be found in the <a href="https://ai.google/static/documents/palm2techreport.pdf">technical report</a>. Using the Vertex AI SDK, you can easily call the publisher endpoints for this model:</p><pre>vertexai.init(project=PROJECT_ID, location=LOCATION)<br>model = TextGenerationModel.from_pretrained(&quot;text-bison@001&quot;)<br>model.predict(<br>    prompt,<br>    max_output_tokens=max_output_tokens, # default 128<br>    temperature=temperature, # default 0<br>    top_p=top_p, # default 1<br>    top_k=top_k) # default 40</pre><h3>User-managed service account for Cloud Run</h3><p>Since the application is deployed in Cloud Run, it uses the permissions of the compute service account by default to call the model. It’s recommended to use a separate service account <strong>with the minimum permissions</strong>. To do that, <a href="https://cloud.google.com/run/docs/securing/service-identity">create a service account with impersonation</a> and the following two extra roles: roles/aiplatform.user to be able to call predictions and roles/logging.logWriter to be able to write logs:</p><pre># Create service account<br>gcloud iam service-accounts create cloud-run-llm \<br>    --description=&quot;Service account to call LLM models from Cloud Run&quot; \<br>    --display-name=&quot;cloud-run-llm&quot;<br><br># add aiplatform.user role<br>gcloud projects add-iam-policy-binding argolis-rafaelsanchez-ml-dev \<br>    --member=&quot;serviceAccount:cloud-run-llm@argolis-rafaelsanchez-ml-dev.iam.gserviceaccount.com&quot; \<br>    --role=&quot;roles/aiplatform.user&quot;<br><br># add logging.logWriter role<br>gcloud projects add-iam-policy-binding argolis-rafaelsanchez-ml-dev \<br>    --member=&quot;serviceAccount:cloud-run-llm@argolis-rafaelsanchez-ml-dev.iam.gserviceaccount.com&quot; \<br>    --role=&quot;roles/logging.logWriter&quot;<br><br># add permission to impersonate the sa (iam.serviceAccounts.actAs), since this is a user-namaged sa<br>gcloud iam service-accounts add-iam-policy-binding \<br>    cloud-run-llm@argolis-rafaelsanchez-ml-dev.iam.gserviceaccount.com \<br>    --member=&quot;user:&lt;REPLACE_WITH_YOUR_USER_ACCOUNT&gt;&quot; \<br>    --role=&quot;roles/iam.serviceAccountUser&quot;</pre><h3>Build and deploy in Cloud Run</h3><p>To build and deploy the <a href="https://gradio.app/">Gradio app</a> in <a href="https://cloud.google.com/run/docs/quickstarts/deploy-container">Cloud Run</a>, you need to build the docker in Artifact Registry and deploy it in Cloud Run.</p><p>Note the --allow-unauthenticated parameter (no authentication required to access the app) and the --service-account parameter pointed to the one configured earlier:</p><pre>gcloud auth configure-docker europe-west4-docker.pkg.dev<br>gcloud builds submit --tag europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/genai-text-demo<br>gcloud run deploy genai-text-demo --port 7860 --image europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/genai-text-demo --service-account=cloud-run-llm@argolis-rafaelsanchez-ml-dev.iam.gserviceaccount.com --allow-unauthenticated --region=europe-west4 --platform=managed  --project=argolis-rafaelsanchez-ml-dev</pre><h3>Conclusions</h3><p>This post shows how to deploy a simple Gradio app that exposes a PaLM-2 model for text generation deployed in Cloud Run.</p><p>text-bison@001 use cases includes dialog summarization, text generation, scoring for marketing, and many others.</p><p>You can find the repo with all the code in <a href="https://github.com/rafaelsf80/genai-vertex-text">this link</a>.</p><h3>References</h3><p>[1] PaLM-2 <a href="https://ai.google/static/documents/palm2techreport.pdf">technical report</a><br>[2] YouTube video: <a href="https://youtu.be/Q1zF9pF6flw">Generative AI on Google Cloud</a><br>[3] YouTube video: <a href="https://youtu.be/yg2yHIKQ7oM">Build, tune, and deploy foundation models with Vertex AI</a><br>[4] YouTube video: <a href="https://www.youtube.com/watch?v=-2rQ_AcQMF8">Build, tune, and deploy foundation models with Generative AI Support in Vertex AI</a><br>[5] <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview">Overview</a> of Generative AI support on Vertex AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54e8a398b24b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/generative-ai-palm-2-model-deployment-with-cloud-run-54e8a398b24b">Generative AI — PaLM-2 model deployment with Cloud Run</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-tuning FLAN-T5 XXL with DeepSpeed and Vertex AI]]></title>
            <link>https://medium.com/google-cloud/fine-tuning-flan-t5-xxl-with-deepspeed-and-vertex-ai-af499daf694d?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/af499daf694d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 18:24:05 GMT</pubDate>
            <atom:updated>2023-05-19T06:37:52.862Z</atom:updated>
            <content:encoded><![CDATA[<p>In a <a href="https://medium.com/google-cloud/deploy-flan-t5-xxl-on-vertex-ai-prediction-579953afdc88">previous post</a>, I wrote about how to deploy Flan-T5 XXL in Vertex AI Prediction. This post shows how to <strong>fine-tune a FLAN-T5 XXL model</strong> (11B parameters) with Vertex AI Training. The model will be fine-tuned using a <a href="https://cloud.google.com/compute/docs/accelerator-optimized-machines#a2_vms">a2-highgpu-8g</a> (680 GB RAM, 96 vCPU) machine with 8xA100 GPUs, and the <a href="https://github.com/microsoft/DeepSpeed">DeepSpeed library</a>.</p><p>The fine-tuning of a model of this size <strong>can take one or more days</strong>. For this example the fine-tuning can cost around <strong>USD 720 per day</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*bGxfwbUlOF-kHzCS1pjLKw.gif" /><figcaption>Fig.1 How Flan works. Source. <a href="https://ai.googleblog.com/2021/10/introducing-flan-more-generalizable.html">Google blog</a></figcaption></figure><h3>The dataset: CNN Dailymail</h3><p>This post will use the <a href="https://huggingface.co/datasets/cnn_dailymail">CNN dailymail dataset</a>, which contains news summarization.</p><p>The dataset is preprocessed by running the script preprocessing/create_flan_t5_cnn_dataset.py, which will tokenize the dataset, split the dataset into training and eval, and save it to disk.</p><p>The tokenized datasets will then be copied to a custom docker image required for Vertex AI Training.</p><h3>The model: Flan-T5 XXL</h3><p>If you are familiar with T5, an open-source LLM model from Google, Flan-T5 outperforms T5 at almost everything. For the same number of parameters, Flan-T5 models have been fine-tuned on more than 1000 additional tasks covering <strong>more languages</strong>.</p><p>Flan-T5 XXL is released with <a href="https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints">different sizes</a>: <strong>Small, Base, Large, XL</strong> and <strong>XXL</strong>. XXL is the biggest version of Flan-T5, containing 11B parameters. Original checkpoints from Google for all sizes are available <a href="https://github.com/google-research/t5x/blob/main/docs/models.md">here</a>.</p><p>The Flan-T5 XXL model card can be accessed <a href="https://huggingface.co/google/flan-t5-xxl">here</a>.</p><h3>DeepSpeed ZeRO</h3><p>DeepSpeed is a <a href="https://www.deepspeed.ai/training/">wrapper library</a> built on top of PyTorch to train advanced deep learning models, like Large Language Models (LLM).</p><p>It performs memory-efficient data parallelism and enables training with model parallelism, doing better than existing frameworks like PyTorch’s Distributed Data Parallel. For example, DeepSpeed can train models with up to 13 billion parameters on a single GPU without Out-of-Memory (OOM) issues, or up to 200 billion parameters with model parallelism on multiple GPUs.</p><p>DeepSpeed reduces the training memory footprint through a novel solution called <a href="https://arxiv.org/abs/1910.02054"><strong>Zero Redundancy Optimizer</strong></a> (ZeRO), which <a href="https://www.deepspeed.ai/tutorials/zero/">partitions model states and gradients across accelerators</a>, implemented as incremental stages of optimization:</p><ul><li><strong>Stage 1:</strong> optimizes states.</li><li><strong>Stage 2:</strong> optimizes states and gradients.</li><li><strong>Stage 3:</strong> optimizes states and gradients and weights. Additionally, this stage also enables CPU-offload to CPU for extra memory savings when training larger models.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A4-r-0KLl70ZBTM77-opCw.png" /><figcaption>Fig.2 DeepSpeed ZeRO (Source: <a href="https://www.deepspeed.ai/2021/03/07/zero3-offload.html">deepspeed.ai documentation</a>)</figcaption></figure><p>Since Flan-T5 XXL weights are around 45 GiB, without even considering optimizer states or gradients, we must use model parallelism with a framework like DeepSpeed.</p><p>In this example, I will use <strong>stage 3 optimization</strong> without CPU offload, i.e. no offloading of optimizer states, gradients or weights to the CPU.</p><p>The configuration of the deepspeed launcher can be done through constants in the file run_seq2seq_deepspeed.py. Note the file configs/ds_flan_t5_z3_config_bf16.json configures specific parameters related to Stage 3 optimization:</p><pre>MODEL_ID = &quot;google/flan-t5-xxl&quot;   # Model id to use for training, in Hugging face<br>TRAIN_DATASET_PATH = &quot;data/train&quot; # Path to processed dataset <br>TEST_DATASET_PATH  = &quot;data/eval&quot;  # Path to processed dataset <br>EPOCHS = 3<br>PER_DEVICE_TRAIN_BATCH_SIZE = 8 # Batch size to use for training<br>PER_DEVICE_EVAL_BATCH_SIZE = 8  # Batch size to use for testing<br>GENERATION_MAX_LENTH = 129      # Maximum length to use for generation<br>GENERATION_NUM_BEAMS = 4        # Number of beams to use for generation<br>LR = 1e-4                       # Learning rate to use for training<br>SEED = 42                       # Seed use for training<br>DEEPSPEED_FILE = &quot;configs/ds_flan_t5_z3_config_bf16.json&quot; # deepspeed config file<br>GRADIENT_CHECKPOINTING = True   # Whether to use gradient checkpointing<br>BF_16 = True                    # Whether to use bf16</pre><h3>Build Custom Training Container image</h3><p>Now we need to build the custom training image containing the DeepSpeed training code and the tokenized dataset. Note the dataset preprocessing is already done in a previous step.</p><pre>FROM europe-docker.pkg.dev/vertex-ai/training/pytorch-gpu.1-10:latest<br><br>WORKDIR /<br><br>LABEL com.nvidia.volumes.needed=nvidia_driver<br><br># env variables for proper GPU setup<br>ENV PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin<br>ENV NVIDIA_VISIBLE_DEVICES=all<br>ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility<br>ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64<br><br># copy deepspeed config file<br>COPY configs /configs<br><br># copy preprocessed data. Need to run &quot;create_flan_t5_cnn_dataset.py&quot; first<br>COPY data /data<br><br># copy deepspeed launcher<br>COPY run_seq2seq_deepspeed.py run_seq2seq_deepspeed.py<br><br># install dependencies<br>RUN pip3 --timeout=300 --no-cache-dir install torch==1.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html<br>COPY requirements.txt requirements.txt<br>RUN pip install -r requirements.txt<br><br># Sets up the entry point to invoke the trainer with deepspeed<br>ENTRYPOINT [&quot;deepspeed&quot;, &quot;--num_gpus=8&quot;, &quot;run_seq2seq_deepspeed.py&quot;]</pre><p>Push docker image to <strong>Artifact Registry</strong>. This process can take up to one hour:</p><pre>gcloud auth configure-docker europe-west4-docker.pkg.dev<br>gcloud builds submit --tag europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/finetuning_flan_t5_xxl</pre><h3>Fine-tuning</h3><p>The model is ready to be fine-tuned on <strong>Vertex AI with 8xA100 NVIDIA A100 GPU</strong> (40 GB HBM). This code launches a CustomContainerTrainingJob since you have a Docker image with the training package embedded into it. The workload is run in <a href="https://cloud.google.com/vertex-ai/docs/training/custom-training">Vertex AI Training</a> (fine-tuning in our case), which includes an upload of the model to <a href="https://cloud.google.com/vertex-ai/docs/model-registry/introduction">Vertex AI Model Registry</a>.</p><p>The fine-tuning should take 23–25 hours to complete and can cost around 750 USD. Metrics can be accesed in the logs, for this model I got <em>eval_loss</em>: 1.288099765777588, <em>eval_rouge1</em>: 44.6169, <em>eval_rouge2</em>: 21.7342, <em>eval_rougeL</em>: 31.6094 and <em>eval_rougeLsum</em>: 41.466 with 3 epochs.</p><pre>job = aiplatform.CustomContainerTrainingJob(<br>    display_name=&quot;flant5xxl_deepspeed_&quot; + TIMESTAMP,<br>    container_uri=TRAIN_IMAGE,<br>)<br><br>model = job.run(<br>    replica_count=1,<br>    service_account = SERVICE_ACCOUNT,<br>    tensorboard = TENSORBOARD_RESOURCE,<br>    boot_disk_size_gb=600,<br>    machine_type=&quot;a2-highgpu-8g&quot;,<br>    accelerator_type=&quot;NVIDIA_TESLA_A100&quot;,<br>    accelerator_count = 8,<br>)</pre><h3>Summary</h3><p>This post summarizes how to fine-tune a <a href="https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16">Flan-T5 XXL</a> in Vertex AI Training. This model has a size of 45 GiB and has been fine-tuned with 8xA100 GPU.</p><p>You can find the full code in <a href="https://github.com/rafaelsf80/vertex-flant5xxl-multitask-finetuning">this repo</a>.</p><p>I would like to thank Camus Ma for comments and contributions to this post.</p><h3>References</h3><p>[1] Medium post: <a href="https://medium.com/google-cloud/finetuning-flan-t5-base-and-online-deployment-in-vertex-ai-bf099c3a4a86">Fine-tuning Flan-T5 Base and online deployment in Vertex AI</a><br>[2] Medium post: <a href="https://medium.com/google-cloud/deploy-flan-t5-xxl-on-vertex-ai-prediction-579953afdc88">Deploy FLAN-T5 XXL in Vertex AI Prediction</a><br>[3] Google blog: <a href="https://ai.googleblog.com/2023/02/the-flan-collection-advancing-open.html">The Flan Collection: Advancing open source methods for instruction tuning</a><br>[4] Research paper: <a href="https://arxiv.org/abs/2210.11416.pdf">FLAN-T5</a><br>[5] Original <a href="https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints">FLAN-T5 Checkpoints</a> from Google<br>[6] Phil Schmid blog: <a href="https://www.philschmid.de/fine-tune-flan-t5-deepspeed">Fine-tune FLAN-T5 XL/XXL using DeepSpeed &amp; Hugging Face Transformers</a><br>[7] DeepSpeed <a href="https://www.deepspeed.ai/training/">training documentation</a></p><p><strong>[Looking for the latest Google Cloud generative AI news? Check out </strong><a href="https://cloud.google.com/blog/transform/prompt-debunking-five-generative-ai-misconceptions"><strong>The Prompt</strong></a><strong> on </strong><a href="https://cloud.google.com/transform"><strong>Transform with Google Cloud</strong></a><strong>.]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af499daf694d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/fine-tuning-flan-t5-xxl-with-deepspeed-and-vertex-ai-af499daf694d">Fine-tuning FLAN-T5 XXL with DeepSpeed and Vertex AI</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy Flan-T5 XXL on Vertex AI Prediction]]></title>
            <link>https://medium.com/google-cloud/deploy-flan-t5-xxl-on-vertex-ai-prediction-579953afdc88?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/579953afdc88</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 12:09:53 GMT</pubDate>
            <atom:updated>2023-03-10T05:20:29.128Z</atom:updated>
            <content:encoded><![CDATA[<p>In a <a href="https://medium.com/google-cloud/finetuning-flan-t5-base-and-online-deployment-in-vertex-ai-bf099c3a4a86">previous post</a>, I wrote about how to fine-tune and deploy a Flan-T5 Base model (220M parameters) in Vertex AI. This post shows how to <strong>deploy a FLAN-T5 XXL model</strong> (11B parameters) in Vertex AI Prediction. The model will be downloaded and embedded in a custom prediction image. You will use a a2-highgpu-1g machine type in <strong>Vertex AI Prediction containing 1xA100 GPU</strong>.</p><p>Note that building the image and deploying it into Vertex AI is a process that <strong>can take up to 4 hours</strong>, and the cost of a Vertex Prediction endpoint (24x7) is <strong>around USD 170 per day</strong> with 1xA100 GPU.</p><p>A demo based on <a href="https://gradio.app/">Gradio</a> and deployed in <a href="https://cloud.google.com/run">Cloud Run</a> is also provided as a quick setup for you to make requests to the deployed model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*rvKHXcgaqRStAkbeuaMwMA.png" /><figcaption>Fig.1 Gradio app on mobile</figcaption></figure><h3>The model: Flan-T5 XXL (fp16)</h3><p>If you are familiar with T5, an open-source LLM model from Google, Flan-T5 improves T5 at almost everything. For the same number of parameters, Flan-T5 models have been fine-tuned on more than 1000 additional tasks covering <strong>more languages</strong>.</p><p>As mentioned in the <a href="https://arxiv.org/abs/2210.11416.pdf">paper</a>:</p><blockquote><em>We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation, RealToxicityPrompts). […] Flan-PaLM achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*bGxfwbUlOF-kHzCS1pjLKw.gif" /><figcaption>Fig.2 How Flan works. Source. <a href="https://ai.googleblog.com/2021/10/introducing-flan-more-generalizable.html">Google blog</a></figcaption></figure><p>Flan-T5 is released with <a href="https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints">different sizes</a>: <strong>Small, Base, Large, XL </strong>and <strong>XXL</strong>. XXL is the biggest version of Flan-T5, containing 11B parameters. Original checkpoints from Google for all sizes are available <a href="https://github.com/google-research/t5x/blob/main/docs/models.md">here</a>.</p><p>The Flan-T5 XXL model card can be accessed <a href="https://huggingface.co/google/flan-t5-xxl">here</a>.</p><p>You will use the <strong>fp16 version</strong> (floating point 16) of the model, which reduces the original size (42 GiB) by half (21 GiB). This reduced version of the model is easier to manage and can be downloaded from <a href="https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16/tree/main">Hugging Face</a>.</p><p>You must <a href="https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16/tree/main">download the model</a> and store it in predict/flan-t5-xxl-sharded-fp16 directory, with a content similar to what is shown below. Note that handler.py is not needed since you will not be using TorchServe:</p><pre>config.json<br>pytorch_model-00001-of-00012.bin<br>pytorch_model-00002-of-00012.bin<br>pytorch_model-00003-of-00012.bin<br>pytorch_model-00004-of-00012.bin<br>pytorch_model-00005-of-00012.bin<br>pytorch_model-00006-of-00012.bin<br>pytorch_model-00007-of-00012.bin<br>pytorch_model-00008-of-00012.bin<br>pytorch_model-00009-of-00012.bin<br>pytorch_model-00010-of-00012.bin<br>pytorch_model-00011-of-00012.bin<br>pytorch_model-00012-of-00012.bin<br>pytorch_model.bin.index.json<br>special_tokens_map.json<br>spiece.model<br>tokenizer.json<br>tokenizer_config.json</pre><h3>Build the Custom Prediction Container image</h3><p>A Custom Container image for predictions is required. A Custom Container image in Vertex AI <a href="https://cloud.google.com/ai-platform-unified/docs/predictions/custom-container-requirements#image">requires that the container runs an HTTP server</a>. Specifically, the container must listen and respond to liveness checks, health checks, and prediction requests.</p><p>You will use the <a href="https://github.com/tiangolo/uvicorn-gunicorn-docker">Tiangolo Uvicorn-Gunicorn</a> web server. It does not have advanced capabilities like NVidia Triton Inference Server or TorchServe about handling multiple models, but enough for the purposes of this demo. You must build and push the container image to <a href="https://cloud.google.com/artifact-registry">Artifact Registry</a>:</p><pre>gcloud auth configure-docker europe-west4-docker.pkg.dev<br>gcloud builds submit --tag europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/flan-t5-xxl-sharded-fp16 --machine-type=e2-highcpu-8 --timeout=&quot;2h&quot; --disk-size=300</pre><p>This build process can take <strong>up to 2 hours</strong> even with a e2-highcpu-8 due to the container size (around 20 GiB).</p><h3>Deploy the model to Vertex AI Prediction</h3><p>Upload and deploy the image to Vertex AI Prediction using the provided script: python3 upload_custom.py. You will use a a2-highcpu-1g (12 vCPU, 85 GB RAM) with 1xA100 (40 GB HBM2) to deploy. Note the <a href="https://cloud.google.com/vertex-ai/pricing#custom-trained_models">price of $4.22 per node-hour</a> plus the <a href="https://cloud.google.com/compute/gpus-pricing">price of $2.9 per hour for the GPU</a>.</p><p>The upload and deploy process <strong>may take up to 2 hours</strong>. Note the parameter deploy_request_timeout to avoid a 504 Deadline Exceeded error during the deployment:</p><pre>DEPLOY_IMAGE = &#39;europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/flan-t5-xxl-uvicorn&#39; <br>HEALTH_ROUTE = &quot;/health&quot;<br>PREDICT_ROUTE = &quot;/predict&quot;<br>SERVING_CONTAINER_PORTS = [7080]</pre><pre>model = aiplatform.Model.upload(<br>    display_name=f&#39;custom-finetuning-flan-t5-xxl&#39;,    <br>    description=f&#39;Finetuned Flan T5 XXL model with Uvicorn and FastAPI&#39;,<br>    serving_container_image_uri=DEPLOY_IMAGE,<br>    serving_container_predict_route=PREDICT_ROUTE,<br>    serving_container_health_route=HEALTH_ROUTE,<br>    serving_container_ports=SERVING_CONTAINER_PORTS,<br>)<br>print(model.resource_name)</pre><pre># Retrieve a Model on Vertex<br>model = aiplatform.Model(model.resource_name)</pre><pre># Deploy model<br>endpoint = model.deploy(<br>     machine_type=&#39;a2-highgpu-1g&#39;,<br>     traffic_split={&quot;0&quot;: 100}, <br>     min_replica_count=1,<br>     max_replica_count=1,<br>     accelerator_type= &quot;NVIDIA_TESLA_A100&quot;,    <br>     accelerator_count=1,<br>     traffic_percentage=100,<br>     deploy_request_timeout=1200,<br>     sync=True,<br>)</pre><h3>Scaling</h3><p>Estimating the resources required to serve a model is important. In this example the Flan-T5 XXL (fp16) model is approximately 20 GiB size and the machine type used for predictions has 12 vCPU, 85 GB RAM, and 40 GB in the GPU. It is recommended to make load testing for production deployments.</p><p>Both Uvicorn-Gunicorn server and Vertex AI Prediction provides several options to scale and handle resources properly:</p><ul><li>Vertex AI Prediction provides <a href="https://cloud.google.com/vertex-ai/docs/predictions/configure-compute#scaling">autoscaling of CPUs/GPUs</a>. In this example, no autoscaling is configured, so you will always use 1 machine.</li><li>The Uvicorn-Gunicorn server allows to <a href="https://fastapi.tiangolo.com/deployment/server-workers/">configure the number of workers</a>. In this example, the number of workers is 1. Given the size of the model, you can also try to use 2 workers and serve 2 instances of the model.</li><li>The Uvicorn-Gunicorn server supports <a href="https://docs.gunicorn.org/en/stable/settings.html#preload-app">pre-loading</a>, which can save RAM resources by preloading the model before workers are forked (in case of using more than 1 worker). In this example pre-loading is not used.</li><li>You can try the 8-bit version of Flan-T5 XXL, but be aware if <a href="https://github.com/huggingface/transformers/issues/20287">the issues</a>.</li></ul><p>Scaling a model of this size is not the main focus of this post, and hence has not been analyzed.</p><h3>Gradio demo UI</h3><p>You are now ready to send predictions to the deployed model. You can simply use the REST API or the Python SDK, but in this demo you will build a simple UI using <a href="https://gradio.app/">Gradio</a>.</p><p>The Gradio app is built on <a href="https://cloud.google.com/run">Cloud Run</a>. You need to build the docker image first, upload the image to <a href="https://cloud.google.com/artifact-registry">Artifact Registry</a> and then deploy it to Cloud Run. The following command runs the last step (Cloud Run deployment):</p><pre>gcloud run deploy flan-t5-xxl-gradio --port 7860 --image europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/flan-t5-xxl-gradio --allow-unauthenticated --region=europe-west4 --platform=managed</pre><p>The image below shows the Gradio app deployed in Cloud Run. You can test the provided examples or try your owns. Make sure to read the instructions provided in the UI main page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HL-h2NyRTLShk69CpsWE0g.png" /><figcaption>Fig.3 Gradio app on desktop</figcaption></figure><h3>Summary</h3><p>This post summarizes how to deploy a <a href="https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16">Flan-T5 XXL</a> fp16 model in Vertex AI Prediction. This model is 20 GiB and requires a 1xA100 GPU.</p><p>You can find the full code in <a href="https://github.com/rafaelsf80/vertex-flant5xxl-multitask">this repo</a>.</p><p>I would like to thank Camus Ma for comments and contributions to this post.</p><h3>References</h3><p>[1] Google blog: <a href="https://ai.googleblog.com/2023/02/the-flan-collection-advancing-open.html">The Flan Collection: Advancing open source methods for instruction tuning</a><br>[2] Research paper: <a href="https://arxiv.org/abs/2210.11416.pdf">FLAN-T5</a><br>[3] Original <a href="https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints">FLAN-T5 Checkpoints</a> from Google<br>[4] Medium post: <a href="https://medium.com/google-cloud/serving-machine-learning-models-with-google-vertex-ai-5d9644ededa3">Serving Machine Learning models with Google Vertex AI</a><br>[5] Phil Schmid blog: <a href="https://www.philschmid.de/deploy-flan-t5-sagemaker">Deploy FLAN-T5 XXL on Amazon SageMaker</a><br>[6] <a href="https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16">Flan-T5 XXL</a> fp16 model</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=579953afdc88" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/deploy-flan-t5-xxl-on-vertex-ai-prediction-579953afdc88">Deploy Flan-T5 XXL on Vertex AI Prediction</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finetuning Flan-T5-Base and online deployment in Vertex AI]]></title>
            <link>https://medium.com/google-cloud/finetuning-flan-t5-base-and-online-deployment-in-vertex-ai-bf099c3a4a86?source=rss-82391cd06de------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf099c3a4a86</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Rafa Sanchez]]></dc:creator>
            <pubDate>Tue, 28 Feb 2023 09:30:27 GMT</pubDate>
            <atom:updated>2023-03-01T07:20:36.586Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fine-tuning Flan-T5 Base and online deployment in Vertex AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*bGxfwbUlOF-kHzCS1pjLKw.gif" /><figcaption>Fig.1 How Flan works. Source. <a href="https://ai.googleblog.com/2021/10/introducing-flan-more-generalizable.html">Google blog</a></figcaption></figure><p>Google has released the checkpoints of several open-source LLM models including BERT, T5 or UL2. Amongst the most interesting in terms of performance and results is <a href="https://ai.googleblog.com/2021/10/introducing-flan-more-generalizable.html">Flan-T5</a>, a variant of T5 that generalises better and outperforms T5 in many NLP tasks.</p><p>This post shows how to <strong>fine-tune a Flan-T5-Base model</strong> for the <strong>SAMSum</strong> dataset (summary of conversations in English) using Vertex AI. In particular, you will use <strong>Vertex AI Training with a 1xA100 GPU</strong> (40 GB HBM) for fine-tuning, and <strong>Vertex AI Prediction</strong> for online predictions.</p><h3>The dataset: SAMSum</h3><p>The <strong>SAMSum dataset</strong> contains about 16k messenger-like conversations with summaries. Conversations were created and written down by linguists <strong>fluent in English</strong>. This use case is similar for example to the conversation summaries feature <a href="https://support.google.com/chat/answer/12918975?hl=en">available in Google Chat</a>.</p><h3>The model: Flan-T5 Base</h3><p><a href="https://arxiv.org/abs/2210.11416">Flan-T5</a> is a variant that outperforms T5 on a large variety of tasks. It is multilingual and uses instruction fine-tuning that, in general, improves the performance and usability of pretrained language models, particularly T5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*khuXoBvTz68IZOE8QF1OCw.png" /><figcaption>Fig.2 T5 model. Source: <a href="https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html">Google blog</a></figcaption></figure><p>Flan-T5 has <a href="https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints">public checkpoints for different sizes</a>. This code sample will use the <a href="https://huggingface.co/google/flan-t5-base">google/flan-t5-base</a> version.</p><h3>Fine-tuning</h3><p>Using libraries from Hugging Face, you will <strong>fine-tune a Flan-T5-Base model</strong> on the <strong>SAMSum dataset for English conversations</strong>. You will use the <a href="https://huggingface.co/google/flan-t5-base">google/flan-t5-base</a>, that will be fine-tuned and moved to Vertex AI Model registry.</p><p>The model is fine-tuned on <strong>Vertex AI with a 1xA100 NVIDIA GPU</strong> (40 GB HBM). The code launches a Training pipeline, a type of Vertex AI job, which executes the following <strong>three steps sequentially</strong>: create a Vertex AI Managed Dataset (not created here because it ‘s downloaded from Hugging Face), run the workload in <a href="https://cloud.google.com/vertex-ai/docs/training/custom-training">Vertex AI Training</a> (fine-tuning in our case), and upload the model to <a href="https://cloud.google.com/vertex-ai/docs/model-registry/introduction">Vertex AI Model Registry</a>.</p><p>The fine-tuning should take around 45 minutes to complete.</p><pre>job = aiplatform.CustomTrainingJob(<br>    display_name=&quot;flan_t5_base_finetuning_gpu_tensorboard&quot;,<br>    script_path=&quot;flant5base_trainer.py&quot;,<br>    requirements=[&quot;py7zr==0.20.4&quot;,<br>                  &quot;nltk==3.7&quot;,<br>                  &quot;evaluate==0.4.0&quot;,<br>                  &quot;rouge_score==0.1.2&quot;, <br>                  &quot;transformers==4.25.1&quot;,<br>                  &quot;tensorboard==2.11.2&quot;,<br>                  &quot;datasets==2.9.0&quot;,<br>                  &quot;google-cloud-storage==2.7.0&quot;],<br>    container_uri=&quot;europe-docker.pkg.dev/vertex-ai/training/pytorch-gpu.1-10:latest&quot;,<br>    model_serving_container_image_uri=&quot;europe-docker.pkg.dev/vertex-ai/prediction/pytorch-gpu.1-10:latest&quot;,<br>)<br><br>model = job.run(<br>    model_display_name=&#39;flan-t5-base-finetuning-gpu-tensorboard&#39;,<br>    replica_count=1,<br>    service_account = SERVICE_ACCOUNT,<br>    tensorboard = TENSORBOARD_RESOURCE_NAME,<br>    machine_type=&quot;a2-highgpu-1g&quot;,<br>    accelerator_type=&quot;NVIDIA_TESLA_A100&quot;,<br>    accelerator_count = 1,<br>)</pre><h3>Uvicorn</h3><p>A Custom Container image for predictions is required. Custom Container image <a href="https://cloud.google.com/ai-platform-unified/docs/predictions/custom-container-requirements#image">requires that the container runs an HTTP server</a>. Specifically, the container must listen and respond to liveness checks, health checks, and prediction requests.</p><p>This repo uses <strong>FastAPI and Uvicorn</strong> to implement the HTTP server. The HTTP server listens for requests on 0.0.0.0. <a href="https://www.uvicorn.org/">Uvicorn</a> is an ASGI web server implementation for Python. Uvicorn currently supports HTTP/1.1 and WebSockets. <a href="https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker">Here</a> is a docker image with Uvicorn managed by Gunicorn for high-performance FastAPI web applications in Python 3.6+ with performance auto-tuning. An uvicorn server is launched with:</p><pre>uvicorn main:app --host 0.0.0.0 --port 8080</pre><h3>Export model from Vertex AI Model Registry</h3><p>After fine-tuning, the model is stored in <a href="https://cloud.google.com/vertex-ai/docs/model-registry/introduction">Vertex AI Model Registry</a>. Since you are going to embed the model in a custom Vertex Prediction Container, you must export the model and place it in the local Prediction Container directory predict/model-output-flan-t5-base directory (for example, with gcloud storage cp). The predict/ directory contains the Dockerfile to generate the Custom Prediction Container image.</p><p>The model must be available in the predict/model-output-flan-t5-base directory, with a similar content like this, before building the custom container prediction image. Note both model and tokenizer must be included:</p><pre>config.json<br>pytorch_model.bin<br>source.spm<br>special_tokens_map.json<br>target.spm<br>tokenizer_config.json<br>training_args.bin<br>vocab.json</pre><h3>Build Custom Prediction Container image and upload model to Vertex AI Model Registry</h3><p>Push docker image to <strong>Artifact Registry</strong>:</p><pre>gcloud auth configure-docker europe-west4-docker.pkg.dev<br>gcloud builds submit --tag europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/finetuning_flan_t5_base</pre><p>Next step is to upload and deploy the model to <a href="https://cloud.google.com/vertex-ai/docs/predictions/overview">Vertex AI Prediction</a>. You will use the previous image with this sample code. Note we are not using GPU for prediction:</p><pre>DEPLOY_IMAGE = &#39;europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/ml-pipelines-repo/finetuning_flan_t5_base&#39; <br>HEALTH_ROUTE = &quot;/health&quot;<br>PREDICT_ROUTE = &quot;/predict&quot;<br>SERVING_CONTAINER_PORTS = [8080]<br><br>model = aiplatform.Model.upload(<br>    display_name=f&#39;custom-finetuning_flan_t5_base&#39;,    <br>    description=f&#39;Finetuned Flan T5 model with Uvicorn and FastAPI&#39;,<br>    serving_container_image_uri=DEPLOY_IMAGE,<br>    serving_container_predict_route=PREDICT_ROUTE,<br>    serving_container_health_route=HEALTH_ROUTE,<br>    serving_container_ports=SERVING_CONTAINER_PORTS,<br>)<br><br># Retrieve a Model on Vertex<br>model = aiplatform.Model(model.resource_name)<br><br># Deploy model<br>endpoint = model.deploy(<br>      machine_type=&#39;n1-standard-4&#39;, <br>      sync=False<br>)<br>endpoint.wait()</pre><h3>Online predictions</h3><p>Predict using the Vertex AI Python SDK:</p><pre># Retrieve an Endpoint on Vertex<br>print(endpoint.predict([[sample[&quot;dialogue&quot;]]]))<br># Output: <br># Prediction(predictions=[[[&quot;Patti&#39;s cat is fine. Patti will pick her up later. Patti will fetch the cage after work.&quot;]]], <br># deployed_model_id=&#39;4495568794441220096&#39;, model_version_id=&#39;1&#39;, model_resource_name=&#39;projects/989788194604/locations/europe-west4/models/2128205910630203392&#39;, explanations=None)</pre><p>Predict using Vertex AI REST API:</p><pre>curl -X POST -H &quot;Authorization: Bearer $(gcloud auth print-access-token)&quot; -H &quot;Content-Type: application/json&quot; \<br>https://europe-west4-aiplatform.googleapis.com/v1/projects/989788194604/locations/europe-west4/endpoints/7028544517674369024:predict \<br> \<br>-d &quot;{\&quot;instances\&quot;: [\&quot;<br>Greg: Hi Mum, how&#39;s the cat doing?<br>Patti: I just rang the vets, she&#39;s fine!<br>Greg: Thank God, been worrying about her all day!<br>Patti: They said I can pick her up later. I&#39;ll pop home and fetch the cage after work. Should be there at 5ish.<br>Greg: Good, see you at home, bye!<br>\&quot;]}&quot;<br># Output<br>{<br>  &quot;predictions&quot;: [<br>    [<br>      [<br>        &quot;Patti will pick up the cat later. Patti will pop home and fetch the cage after work.&quot;<br>      ]<br>    ]<br>  ],<br>  &quot;deployedModelId&quot;: &quot;4495568794441220096&quot;,<br>  &quot;model&quot;: &quot;projects/989788194604/locations/europe-west4/models/2128205910630203392&quot;,<br>  &quot;modelDisplayName&quot;: &quot;custom-finetuning-flan-t5-base&quot;,<br>  &quot;modelVersionId&quot;: &quot;1&quot;<br>}</pre><h3>Summary and acknowledgements</h3><p>This post summarizes how to fine-tune and deploy a <strong>Flan-T5 Base</strong> model to an online endpoint in Vertex AI.</p><p>It should be noted that this example does not include model parallelism or distributed training, since the model could be considered as “small” (approx. 250M parameters). However, for bigger versions of Flan-T5, like <a href="https://huggingface.co/google/flan-t5-xxl">google/flan-t5-xxl</a>, distributed training will be required.</p><p>You can find the full code <a href="https://github.com/rafaelsf80/vertex-flant5base-summarization">in this repo</a>.</p><p>I would like to thank Camus Ma for comments and contributions to this post.</p><h3><strong>References</strong></h3><p>[1] Google research blog: <a href="https://ai.googleblog.com/2023/02/the-flan-collection-advancing-open.html">The Flan Collection: Advancing open source methods for instruction tuning</a><br>[2] Flan-T5 <a href="https://arxiv.org/abs/2210.11416">paper</a><br>[3] Phil Schmid blog: <a href="https://www.philschmid.de/fine-tune-flan-t5">Fine-tune FLAN-T5 for chat &amp; dialogue summarization</a><br>[4] Vertex AI Training: <a href="https://cloud.google.com/vertex-ai/docs/training/custom-training">Create a model using a custom training</a><br>[5] Vertex AI Prediction: <a href="https://cloud.google.com/vertex-ai/docs/predictions/overview#get_predictions_from_custom_trained_models">Get predictions from custom trained models</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf099c3a4a86" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/finetuning-flan-t5-base-and-online-deployment-in-vertex-ai-bf099c3a4a86">Finetuning Flan-T5-Base and online deployment in Vertex AI</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>