<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Paul Balm on Medium]]></title>
        <description><![CDATA[Stories by Paul Balm on Medium]]></description>
        <link>https://medium.com/@pbalm?source=rss-94fa7ea880d1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Yj6aJ6Lfoo14UYz2</url>
            <title>Stories by Paul Balm on Medium</title>
            <link>https://medium.com/@pbalm?source=rss-94fa7ea880d1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:41:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pbalm/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to prompt Gemini asynchronously using Python on Google Cloud]]></title>
            <link>https://medium.com/google-cloud/how-to-prompt-gemini-asynchronously-using-python-on-google-cloud-986ca45d9f1b?source=rss-94fa7ea880d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/986ca45d9f1b</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[gcp-app-dev]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Paul Balm]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 16:40:23 GMT</pubDate>
            <atom:updated>2024-09-23T20:02:01.532Z</atom:updated>
            <content:encoded><![CDATA[<p>Gemini is a family of multimodal large language models, developed by Google DeepMind. They are currently (June 2024) Google’s most capable models of this type. Gemini is also the name of a <a href="https://gemini.google.com/">consumer facing web application</a> that uses the Gemini models, similar to that other one that people talk about, but here I talk about the models.</p><p>In this post I’d like to show a solution to a common problem that occurs when you have a long list of questions or prompts to ask of your model. For example, you may want to evaluate a prompt template with a set of questions and compare them to a set of answer.</p><p>The problem here is that running your test set of questions through the model can take a long time, if you send them one by one. It would be much more efficient if you could send all questions at once, and then wait for all the answers to come back. This is called “prompting the model asynchronously”.</p><p>Here’s how you can prompt Gemini 1.5 pro version 001 in a <em>synchronous</em> manner:</p><pre>import vertexai<br>from vertexai.generative_models import GenerativeModel, Part, FinishReason<br>import vertexai.preview.generative_models as generative_models<br><br>def generate(prompt, my_project):<br>  vertexai.init(project=my_project, location=&quot;us-central1&quot;)<br>  model = GenerativeModel(<br>    &quot;gemini-1.5-pro-001&quot;,<br>  )<br>  responses = model.generate_content(<br>      [prompt],<br>      generation_config=generation_config,<br>      safety_settings=safety_settings,<br>      stream=True,<br>  )<br><br>  return &quot;&quot;.join([response.text for response in responses])<br><br>generation_config = {<br>    &quot;max_output_tokens&quot;: 8192,<br>    &quot;temperature&quot;: 1,<br>    &quot;top_p&quot;: 0.95,<br>}<br><br>safety_settings = {<br>    generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>}<br><br>my_project = &#39;YOUR_PROJECT_HERE&#39;<br>print(generate(&quot;hello&quot;, my_project))</pre><p>The above code requires you to specify “my_project” which would be your Google Cloud project, and that’s where your calls to Gemini are billed. It will probably print something like:</p><p>Hello! 👋 How can I help you today? 😊</p><p>I just tried it and it took about 2 seconds. Now suppose you wanted to learn a few recipes, you could prompt Gemini for this. These are longer responses that take a bit longer to generate: The examples below typically take about 12 seconds each. Now imagine you wanted to learn how to cook your 8 favorite Dutch food items:</p><pre>foodstuffs = [&#39;zure haring&#39;, &#39;stroopwafels&#39;, &#39;hutspot&#39;, &#39;stamppot rauwe andijvie&#39;, &#39;zoute drop&#39;, &#39;poffertjes&#39;, &#39;bitterballen&#39;, &#39;oliebollen&#39;]<br><br>[generate(f&#39;give me a recipe for {f}&#39;) for f in foodstuffs]</pre><p>This will take around 8 times 12: 96 seconds to execute. That may not be a very long time, but if you wanted to get a collection of 100 Dutch recipes, or maybe 100 questions to validate the answers, it would take about 20 minutes. What if we could send these prompt in parallel?</p><p>We will use the packages asyncio and tenacity to define an asynchronous version of the “generate” function, here’s the code:</p><pre>import asyncio<br>from tenacity import retry, wait_random_exponential<br><br>import vertexai<br>from vertexai.generative_models import GenerativeModel, Part, FinishReason<br>import vertexai.generative_models as generative_models<br><br>@retry(wait=wait_random_exponential(multiplier=1, max=120))<br>async def async_generate(prompt, my_project):<br>  vertexai.init(project=my_project, location=&quot;us-central1&quot;)<br>  model = GenerativeModel(<br>    &quot;gemini-1.5-pro-001&quot;,<br>  )<br>  response = await model.generate_content_async(<br>      [prompt],<br>      generation_config=generation_config,<br>      safety_settings=safety_settings,<br>      stream=False,<br>  )<br><br>  return response.text<br><br><br>generation_config = {<br>    &quot;max_output_tokens&quot;: 8192,<br>    &quot;temperature&quot;: 1,<br>    &quot;top_p&quot;: 0.95,<br>}<br><br>safety_settings = {<br>    generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>}</pre><p>With respect to the earlier code:</p><ul><li>We define the generate method with the async keyword so that is can be run asynchronously</li><li>We call “model.generate_content_async” from inside the async_generate function to get actual asynchronous behaviour</li><li>In order to get the response from “model.generate_content_async”, we have to add the “await” keyword when we call it</li><li>We added a “retry” decorator from the tenacity package. If the call to the model fails, this will cause it to try again. This can easily happen since your project will have a quota to prevent abuse and cost overruns, that limits the number of calls per minute that you can make to the Gemini model, and what we want to do is to send multiple (many?) calls, without waiting for each of the answer first. This specific code will retry a random amount of time, so that if multiple calls hit the quota at the same time, they are not also retried at the exact same time, which would probably just hit the quota again. The wait time is drawn from an exponential distribution between 1 second and 2 minutes.</li></ul><p>We can now use this function get a recipe for poffertjes. From a Jupyter notebook you can do:</p><pre>my_project = &#39;YOUR_PROJECT_HERE&#39;<br>response = await async_generate(&quot;give me a recipe for poffertjes&quot;, my_project))<br>print(response)</pre><p>But if you are executing this code as a stand-alone Python program, you need to start the asyncio event loop yourself (in a notebook, Jupyter has done it for you):</p><pre>my_project = &#39;YOUR_PROJECT_HERE&#39;<br>response = asyncio.run(async_generate(&quot;give me a recipe for poffertjes&quot;, my_project))<br><br>print(response)</pre><p>In either case, this takes again about 12 seconds, so you can expect 96 seconds for 8 recipes. But what we would like to do, is to request all recipes in parallel, which you can do as follows (in a Jupyter notebook):</p><pre>my_project = &#39;YOUR_PROJECT_HERE&#39;<br>get_responses = [async_generate(f&#39;give me a recipe for {f}&#39;, my_project) for f in foodstuffs]<br>recipes = await asyncio.gather(*get_responses)</pre><p>And this will take about 17 seconds to get all your recipes. And again, outside of the Jupyter environment, you will have to start the event loop yourself:</p><pre>async def main():<br>    foodstuffs = [&#39;raw herring&#39;, &#39;stroopwafels&#39;, &#39;hutspot&#39;, &#39;stamppot rauwe andijvie&#39;, &#39;zoute drop&#39;, &#39;poffertjes&#39;, &#39;bitterballen&#39;, &#39;oliebollen&#39;]<br>    get_responses = [async_generate(f&#39;give me a recipe for {f}&#39;, my_project) for f in foodstuffs]<br>    return await asyncio.gather(*get_responses)<br><br>responses = asyncio.run(main())<br>for r in responses:<br>    print(r)</pre><p>Enjoy!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nYBuLajI-Lf3_5k3ukTFjQ.jpeg" /></figure><h3>Troubleshooting</h3><p>The first thing to note is that this program should respond in about 20 seconds. If nothing happened after one minute, something is wrong! It’s not going to get better by waiting much longer, I’m afraid.</p><p>The reason the program can take a long time, is that we have added an automatic retry loop. This is really useful when you send lots of prompts and you hit your quota of queries per minute. But if there is a bug and all prompts fail, you’re just waiting a long time for all the retries and variable wait times to complete.</p><p>Therefore, if your program doesn’t seem to respond, comment or remove the line that starts with “@retry(…”. That way, each prompt will only be tried once, and if it fails, you will see an error.</p><p>Here’s the full version of the asynchronous program (but you still have to replace “YOUR PROJECT HERE” with your project ID):</p><pre>import asyncio<br>from tenacity import retry, wait_random_exponential<br><br>import vertexai<br>from vertexai.generative_models import GenerativeModel, Part, FinishReason<br>import vertexai.generative_models as generative_models<br><br>#@retry(wait=wait_random_exponential(multiplier=1, max=120))<br>async def async_generate(prompt, my_project):<br>  vertexai.init(project=my_project, location=&quot;us-central1&quot;)<br>  model = GenerativeModel(<br>    &quot;gemini-1.5-pro-001&quot;,<br>  )<br>  response = await model.generate_content_async(<br>      [prompt],<br>      generation_config=generation_config,<br>      safety_settings=safety_settings,<br>      stream=False,<br>  )<br><br>  return response.text<br><br>generation_config = {<br>    &quot;max_output_tokens&quot;: 8192,<br>    &quot;temperature&quot;: 1,<br>    &quot;top_p&quot;: 0.95,<br>}<br><br>safety_settings = {<br>    generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>    generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,<br>}<br><br><br>my_project = &#39;YOUR PROJECT HERE&#39;<br><br>async def main():<br>    foodstuffs = [&#39;raw herring&#39;, &#39;stroopwafels&#39;, &#39;hutspot&#39;, &#39;stamppot rauwe andijvie&#39;, &#39;zoute drop&#39;, &#39;poffertjes&#39;, &#39;bitterballen&#39;, &#39;oliebollen&#39;]<br>    get_responses = [async_generate(f&#39;give me a recipe for {f}&#39;, my_project) for f in foodstuffs]<br>    return await asyncio.gather(*get_responses)<br><br>responses = asyncio.run(main())<br>for r in responses:<br>    print(r)</pre><p>If you save this file to “test.py” then you:</p><ul><li>authenticate using gcloud auth application-default login</li><li>execute the program using python test.py</li></ul><p>Let me know in the comments if you’re still stuck and I’ll try to respond. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=986ca45d9f1b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/how-to-prompt-gemini-asynchronously-using-python-on-google-cloud-986ca45d9f1b">How to prompt Gemini asynchronously using Python on Google Cloud</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Redacting airline PNRs and other PII from Dialogflow CX conversations]]></title>
            <link>https://medium.com/google-cloud/redacting-airline-pnrs-and-other-pii-from-dialogflow-cx-conversations-890804120b87?source=rss-94fa7ea880d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/890804120b87</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[dialogflow-cx]]></category>
            <category><![CDATA[gcp-app-dev]]></category>
            <category><![CDATA[pii]]></category>
            <category><![CDATA[dlp]]></category>
            <dc:creator><![CDATA[Paul Balm]]></dc:creator>
            <pubDate>Sun, 04 Feb 2024 15:19:45 GMT</pubDate>
            <atom:updated>2024-02-05T06:01:10.718Z</atom:updated>
            <content:encoded><![CDATA[<p>As the <a href="https://cloud.google.com/dialogflow?hl=en">documentation</a> states, “Dialogflow is a comprehensive platform for developing chatbots, voice bots, and virtual agents using natural language understanding and Google AI.” And it’s integrated into Google Cloud Platform.</p><p>A frequent challenge when implementing chatbots is that the user can enter sensitive information. Best practice is to log conversations for subsequent analysis, for example to track abandonment rates and to understand user satisfaction. But unless we explicitly scrub the sensitive data, the PII (personally identifiable information) will be logged along with the rest of the conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mJfZK170r-li8RowGSaS8g.png" /></figure><p>Google Cloud’s service for the redaction, encryption or other types of masking, of sensitive data, is called “<a href="https://cloud.google.com/security/products/sensitive-data-protection">Sensitive Data Protection</a>”, or “SDP”. We will use SDP in this article to redact airline PNRs (also often called “booking codes” or “record locator”) and, as an example, first and last names, from conversation data. The redaction will happen before the conversion is written to disk.</p><h3>Logging Conversations</h3><p>Cloud Logging is the service that centralizes the logs of Google Cloud products. Conversation can be logged easily to Cloud Logging, just by enabling “conversation history” as well as “Cloud Logging” in your agent settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wyj7W9piJJrkJeXWLP-M0w.png" /></figure><p>We can also log our conversations to BigQuery, which is more convenient for typical analytics workflows. We will first have to create the destination table (see the <a href="https://cloud.google.com/dialogflow/cx/docs/concept/export-bq#config">documentation</a>):</p><pre>CREATE TABLE &lt;your_dataset_name&gt;.dialogflow_bigquery_export_data(<br>  project_id STRING,<br>  agent_id STRING,<br>  conversation_name STRING,<br>  turn_position INTEGER,<br>  request_time TIMESTAMP,<br>  language_code STRING,<br>  request JSON,<br>  response JSON,<br>  partial_responses JSON,<br>  derived_data JSON,<br>  conversation_signals JSON,<br>  bot_answer_feedback JSON<br>);</pre><p>After this, we just enable BigQuery export in the Agent settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VRkK4oRGighYf6ffJuOvzQ.png" /></figure><p>Note that even if you don’t want to send your conversations to Cloud Logging, you still need to “enable conversation history”, to be able to send it to BigQuery.</p><h3>Defining what data to redact</h3><p>Sensitive Data Protection uses two types of templates to define what to do with sensitive data. The first is the inspection template, which defines what data shall be considered “sensitive”. And the second is the “de-identification template”, which defines the actions to apply to the sensitive data.</p><p>In Dialogflow, we <em>only</em> use the inspection template, and Dialogflow handles the de-identification. SDP is very flexible in applying different kinds of treatments, such a reversible or non-reversible encryption, format-preserving encryption, and more. But Dialogflow can simply apply “masking”, which is to overwrite the sensitive data with the string “[redacted]”, and therefore, we don’t need the “de-identification template” from SDP if this is sufficient for our use case.</p><p>We said we wanted to redact names and airline PNRs. SDP calls these “info types”. We need to create an inspection template with info types for names and airline PNRs. On the Configuration tab of SDP, we will find a button to create a template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8IhorjthffZgpzlKJcotmg.png" /></figure><p>The UI to create templates has a section for basic information, such as the template name, and a configuration section. For the template name, I like to use something that tells me what it does, so I will use “pnrs_and_names”. In the configuration section, we’re going to configure the info types that this template should flag. SDP has many info types built in, and since we wanted to redact names, we’ll choose the predefined info type <em>PERSON_NAME</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2EtnH8J8IDDmKgJb3SVgTg.png" /></figure><p>But there is no predefined info type for “airline PNR” or “airline booking code”. We’re going to have to use a custom info type for that. There are several types of custom info types, such as those backed by a dictionary in BigQuery, or a short list of words (a small, inline dictionary). Here, we will use the “regex” option, with a regular expression, that matches an airline PNR. An airline PNR contains capital letters and digits, 5 characters when it refers to a reservation in the airline system and 6 characters when it refers to a reservation in a distribution system such as Amadeus or Sabre. A regex to match these patterns is <em>[A-Z0–9]{5,6}</em>. The following screenshot shows how to configure the “PNR” info type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAFN2CARnqu51PbIdYE8zg.png" /></figure><p>If we add this info type to our inspection template, we are all set! You may want to tweak the “likelihood” which indicates the probability that a string matching the regex is actually a PNR and the sensitivity of PNR data — a PNR is generally considered a personal data point, but not a very sensitive one like a phone number, email address, or credit card number might be.</p><p>Just one more thing: After saving your template, you will see the list of templates in your project. Click on the template that you just created, and note the fully qualified ID. In my case, it says <em>“projects/[my project here]/locations/global/inspectTemplates/pnrs_and_names”</em>.</p><h3>Configure Sensitive Data Treatment in Dialogflow</h3><p>I have a basic agent, that I just created for testing. The sensitive data treatment is under Agent Settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fRIY4RMWTZYUFfKHZY2mkA.png" /></figure><p>On the Security tab of the agent setting, we need to select the security settings that will scrub names and PNRs, but we will first have to create these settings by clicking “Manage Security Settings”. This takes you to the Contact Center AI page. CCAI is the group of products and services to support contact centersin Google Cloud, of which Dialogflow is an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dC509R2N-RFYh5aGrLf-Bg.png" /></figure><p>The <a href="https://cloud.google.com/dialogflow/cx/docs/concept/security-settings">documentation</a> will tell you how to configure the security settings, or if you feel adventurous, you can copy them fom the above screenshot. Under “inspect template”, we specify the fully qualified template ID that we noted earlier. If you save the security settings, you can now select these settings in the Agent settings in Dialogflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v6a3xjb2qiPO64hwQ-SXsw.png" /></figure><h3>Ready to test!</h3><p>Since my bot doesn’t have any specific configuration, it won’t be very helpful, but we can use it for testing. Here we go, using the “test agent” functionality of Dialogflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*MkgJd-uzT8KvU2qPphbDuw.png" /></figure><p>I have configured the agent to log the conversation to BigQuery. In the BigQuery export, each row contains a user input and the corresponding bot response. This information is contained in a JSON document for the user input, and another one for the bot response. If we just want to see what the user said, and what the bot said, we will have to extract this from the JSON, using BigQuery’s JSON processing functions. I use the following query:</p><pre>SELECT<br>  SPLIT(conversation_name, &#39;/&#39;)[7] as session_id, <br>  turn_position, <br>  request_time, <br>  JSON_QUERY(request, &#39;$.queryInput.text.text&#39;) as query, <br>  JSON_QUERY(derived_data, &#39;$.agentUtterances&#39;) as response<br>FROM `MY_PROJECT_HERE.dialogflow_history.dialogflow_bigquery_export_data` <br>WHERE DATE(request_time) = CURRENT_DATE()<br>ORDER BY conversation_name, turn_position</pre><p>Aside from extracting the user query and bot response, I extract the session ID and turn position, to keep conversations together and in the correct order. The result of this query is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfuYirdYO7KfRsa_ZVe5wg.png" /></figure><p>You can see that the second user input is logged as “My name is [redacted] and my booking code in [redacted]. I would like to change my flight” (typo included). Success!</p><h3>Conclusion</h3><p>I hope this quick write-up of a specific and concrete example is useful. If you’re looking for a more comprehensive description of the functionalities, I recommend “<a href="https://cloud.google.com/blog/topics/developers-practitioners/redacting-pii-data-dialogflow-cx-google-cloud-data-loss-prevention-dlp">Redacting PII data in Dialogflow CX with Google Cloud Data Loss Prevention (DLP)</a>” on the Google Cloud blog. (Note that the Data Loss Prevention service has recently been rename “Sensitive Data Protection”.) Happy conversing!</p><h3>References</h3><ol><li><a href="https://cloud.google.com/dialogflow?hl=en">Dialogflow product description</a></li><li><a href="https://cloud.google.com/security/products/sensitive-data-protection?hl=en">Sensitive Data Protection service documention</a></li><li>Dialogflow documentation: <a href="https://cloud.google.com/dialogflow/cx/docs/concept/export-bq#config">Log conversations to BigQuery</a></li><li><a href="https://cloud.google.com/dialogflow/cx/docs/concept/security-settings">CCAI Security Settings documentation</a></li><li>Google Cloud Blog: <a href="https://cloud.google.com/blog/topics/developers-practitioners/redacting-pii-data-dialogflow-cx-google-cloud-data-loss-prevention-dlp">Redacting PII data in Dialogflow CX with Google Cloud Data Loss Prevention (DLP)</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=890804120b87" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/redacting-airline-pnrs-and-other-pii-from-dialogflow-cx-conversations-890804120b87">Redacting airline PNRs and other PII from Dialogflow CX conversations</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Pipeline Development on Google Cloud]]></title>
            <link>https://medium.com/google-cloud/machine-learning-pipeline-development-on-google-cloud-5cba36819058?source=rss-94fa7ea880d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cba36819058</guid>
            <category><![CDATA[kubeflow-pipelines]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[vertex-ai-pipelines]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Paul Balm]]></dc:creator>
            <pubDate>Sat, 27 Jan 2024 15:19:36 GMT</pubDate>
            <atom:updated>2024-02-07T18:16:41.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>Enterprise MLOps part 2: ML Pipelines with Vertex AI</h3><p>Many organizations using Vertex AI are working on operationalizing their machine learning work using Google Cloud infrastructure, so that they can scale their work and expand the impact of ML in their business.</p><p>Operationalization requires automation and orchestration of the different tasks involved: data loading, preprocessing, model training, evaluation… And we typically implement these tasks as a pipeline.</p><p>It turns out that deploying a series of tasks that are orchestrated to run in collaboration with each other on remote cloud infrastructure involves a steep learning curve. This technology is not easy to learn, and building a good pipeline requires a number of design decisions. In this post, we will describe what the guiding principles that we use across our projects at <a href="https://cloud.google.com/consulting">Google Cloud Consulting</a>.</p><p>We will focus our examples on <a href="https://www.tensorflow.org/tfx">TFX</a> and <a href="https://www.kubeflow.org/docs/components/pipelines/v2/introduction/">Kubeflow Pipelines</a>, because these are the pipelines that can be executed natively on Google Cloud using <a href="https://cloud.google.com/vertex-ai/docs/pipelines/introduction">Vertex AI Pipelines</a>, Google Cloud’s fully managed ML pipeline service. It’s possible to execute many other types of pipelines on Google Cloud, for example using Spark or Airflow. But Vertex AI Pipelines is a pipeline orchestrator that provides some valuable features for machine learning.</p><p>We also limit our focus on ML pipelines that take (training) data as an input and have a model as their primary output. These are not the only ML pipelines, because other processes that are part of an ML system can be implemented as pipelines, and can therefore be considered “ML pipelines” as well. Examples could be a monitoring pipeline and an evaluation pipeline. But we focus here on the “ML training pipeline”.</p><p>This is the second article in our series on MLOps processes:</p><ul><li>Part 1: <a href="https://medium.com/google-cloud/enterprise-mlops-with-google-cloud-vertex-ai-part-1-9f36211ea66c">Google Cloud Infrastructure setup</a></li><li>Part 2: ML Pipelines with Vertex AI (this article)</li><li>Part 3: <a href="https://jgpuga.medium.com/enterprise-mlops-with-google-cloud-vertex-ai-part-3-ci-cd-33d5e6e774a7">CI/CD for MLOps on Vertex AI</a></li></ul><p>But before we get started, let’s take a step back and consider:</p><h3>Why pipelines?</h3><p>The problem at hand is that to operationalize the process of training machine learning models. Training a model requires loading training data to where the model is being trained, for example by extracting it from a database. Before training the model, we probably want to apply some preprocessing: Data cleaning and imputation, feature engineering, and data validation are typical activities. We then want to train the model, evaluate its performance, deploy it… In short, we have a large number of activities that need to be executed in a certain order.</p><p>“Pipeline” is a design pattern. It’s a way of implementing an application that runs tasks in a certain order. But at least in simple cases, we can use a shell script to do this, which is not hard to learn at all. So why use the pipeline pattern?</p><p>For some applications, the “monolith”, where all the code is in one application that runs on one computer, is a perfectly good design. Even if our application trains a ML model, if the model is fairly trivial and training requires few resources, then building a monolith can be reasonable. But in real-life ML scenarios, a pipeline is usually a better choice. A pipeline:</p><ul><li>Has a modular structure, which improves maintainability and understandability</li><li>Allows us to choose the right hardware for execute different “modules” (pipeline steps) on different hardware</li><li>Facilitates tracking of intermediate artifacts, which in turn helps debugging, monitoring, and also efficiency because it permits caching intermediate results and restarting from previously cached results.</li></ul><h3>What’s a good pipeline?</h3><p>Without trying to be exhaustive, let’s review some attributes of good pipelines.</p><p>First, a good pipeline achieves the right level of modularity. When we organize the work that needs to be done to train a new model (and after training it), we split it into different pipeline steps. What is the right “size” of each step? When should we continue to split, and when are we done? One should consider efficiency: Larger steps tend to be more efficient with the hardware, because there is less hand-off between steps, but they are harder to parallelize. Breaking down your application into different steps can make the application easier to understand when it executes, but if you make very many very small blocks, then it can actually become harder to understand. Similarly for reusability of pipeline components: If it’s too small, there is little value and if it’s too large, it becomes unwieldy and difficult to reuse for that reason. Another consideration is that you may want to track the output of each block, but if the blocks are too small, the output may not be meaningful or easy to understand. In summary, there are no hard rules, but important considerations are efficiency, understandability, reusability and traceability.</p><p>It’s also important for a pipeline to be complete in terms of functionality. If we work back from the nuclear task that is training the model: Preparing the training data should not involve manual steps that need to be performed before the pipeline can be started. Looking at what comes after training: The pipeline needs to leave the model (weights) where it can be used. This can mean, for example, uploading to a <a href="https://cloud.google.com/vertex-ai/docs/model-registry/introduction">model registry</a>, or deploying it, or both.</p><p>Aside from uploading the model and leaving it ready to use, the pipeline should prepare metadata or documentation that gives us the information that we need to use this model confidently, in terms of performance, legal aspects, and more.</p><p>Completeness also means having functionality that a robust ML system requires. If we want to implement full “continuous deployment” (the often overlooked half of CI/CD), then we should deploy our model from our pipeline. A robust deployment implements monitoring that allows us to track the model performance, so we can know when something goes wrong, or when gradual degradation has reached the point where we want to retrain our model with the latest data.</p><h3>Pipeline frameworks: Kubeflow vs. TFX</h3><p>The Kubeflow Pipelines SDK provides different types of components and intermediate artifacts, and allows you to use these to construct pipelines. For example, you can use a “light-weight Python component” that transforms a Python function into a step in your pipeline. The return values from this component can be passed to the next pipeline component.</p><figure><img alt="Kubeflow logo" src="https://cdn-images-1.medium.com/max/300/0*uSlYiTNsPoKPw1zi.png" /></figure><p>This is very different from how TFX works. TFX is short for Tensorflow Extended. Originally, it’s built around Tensorflow, which allows you to define and train models, but nothing else. TFX is a collection of pipeline components that you configure. These pipeline components fit together in one way, and you configure them to do what you want. For example, you will have to provide the source for your training data so that the process can start, and you will have to provide the training script for the component that trains the model. Some components are optional, but basically, TFX provides you with one pipeline that always has the same steps in all your ML projects. The advantage is that the TFX pipeline components have already solved many problems that an operational ML system will have to solve. Kubeflow components are easier to understand, but unless you use the reference pipeline that we will describe later, you are essentially starting from scratch. A downside of TFX is that it fits more naturally with Tensorflow models, but you can train any type of model using TFX.</p><figure><img alt="Tensorflow logo" src="https://cdn-images-1.medium.com/max/224/0*nWxEh28iXl_gcyOF.png" /></figure><h3>Best Practices</h3><p>We will start by providing a reference pipeline implemented using Kubeflow, and one using TFX. Then we will follow up these reference implementations with the best practices that they implement.</p><h3>Kubeflow Reference pipeline</h3><p>Our Kubeflow reference pipeline has the following components:</p><ul><li>Load and split data: Extraction of the training data from its storage (for example, BigQuery or an on-prem database or API) and split it into training data, evaluation data and test data</li><li>Model training produces a trained model plus performance metrics</li><li>Upload the trained model to the Vertex Model Registry</li><li>Upload a model evaluation to the Vertex Model Registry:<br>- Upload the evaluation data to BigQuery<br>- Optionally, sample the data down to obtain a reasonable amount for evaluation<br>- Execute a batch prediction job to generate predictions from this sample<br>- Reformat these predictions to match the expectation of the following step<br>- Provide an evaluation by joining the predictions back to the evaluation data sample and comparing them to the ground truth<br>- Import the evaluation to the Model Registry to enrich the model uploaded previously</li><li>Generate a model card:<br>- Generate some plots/graphics (like histograms) using the evaluation data<br>- Combine these graphics with statistics of the training and test data produced earlier to generate an HTML model card</li></ul><p>The <a href="https://github.com/GoogleCloudPlatform/professional-services/tree/main/examples/vertex_mlops_enterprise/src/kfp_pipelines">full code for this pipeline</a> is available in our Github repository.</p><figure><img alt="A flow diagram of our Kubeflow reference pipeline" src="https://cdn-images-1.medium.com/max/921/0*YMPYU041zn4wzKva" /><figcaption>A diagram of our Kubeflow reference pipeline, as displayed by the UI of Vertex AI Pipelines.</figcaption></figure><h3>TFX Reference pipeline</h3><p>The TFX pipeline is constructed from predefined components:</p><ul><li>TrainDataGen loads the training data, for example from BigQuery and transforms it into Tensorflow’s efficient binary “Example” format</li><li>Generate statistics on the training data (min, max, median, number of missing values, etc)</li><li>Import a predefined schema for the training data, including value ranges and dictionaries for categorical variables</li><li>Validate the training data against the imported schema and abort the timeline if the training data does not match expectation. This is especially useful when retraining happens automatically using the latest data.</li><li>Transform the input data to produce the features that we will use for model training. This step also exports a persistent transform operation that we can reuse during serving.</li><li>Train the model:<br>- Retrieve the hyperparameters to be used during training (learning rate, number of epochs, etc). This is convenient when tuning the hyperparameters.<br>- Retrieve a previously trained model to avoid starting from scratch (i.e. use a warm start)<br>- Train the actual model</li><li>To evaluate the trained model:<br>- Identify the baseline model for comparison (this could be the currently deployed model, or best trained model thus far)<br>- Retrieve the baseline model<br>- Evaluate both baseline and currently trained model.</li><li>Depending on the evaluation output, publish the model (in this case, push to Google Cloud Storage)</li></ul><p>We have the <a href="https://github.com/GoogleCloudPlatform/professional-services/tree/main/examples/vertex_mlops_enterprise/src/tfx_pipelines">full source code for this TFX pipeline</a> in our Github repository as well.</p><figure><img alt="A flow diagram of the TFX pipeline." src="https://cdn-images-1.medium.com/max/1024/0*dVrQ6Pucle58WDKU" /><figcaption>A diagram of the TFX pipeline, as displayed by the UI of Vertex AI Pipelines.</figcaption></figure><p>We will now review some of the best practices implemented by these pipelines.</p><h3>Data Ingestion and Split</h3><p>Before starting the training process, the training data (including test and evaluation data) needs to be efficiently accessible to the training process. If the data needs to be retrieved from an API with latency, or a heavily loaded database, then we should copy it over to a low latency, high throughput storage such as Google Cloud Storage. Very large datasets can still be expensive to store, but per TB, storage costs are low. You should consider maintaining a copy of the original dataset, so that you can trace any trained model back to the input that was used. This can be useful for training, but for testing and validation as well.</p><p>Another good practice is to split the input data into the datasets that you require (typically, training, testing and evaluation) and output these splits as separate artifacts of the pipeline step. This allows you to check the dependency diagram of the pipeline, like the diagrams shown before, and ensure that the dataset used to evaluate the model is never used during the training process. Outputting the splits as independent artifacts also allows you to inspect them later, perhaps to understand specific inference results.</p><h3>Generate a model evaluation</h3><p>Communicating the performance characteristics of your models is vital for the confidence in these models when using them. A good ML system will make the evaluation available, together with the model. The evaluation should provide a comprehensive overview of the performance of the model. In order to avoid overhead and mistakes, especially since the purpose is to gain confidence, it’s crucial that this evaluation is generated and published as part of the pipeline, and not through a separate process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9xn45Zrued-ue9Y8" /><figcaption>An example of the UI for model evaluation, as part of Google Cloud’s Vertex AI Model Registry.</figcaption></figure><h3>Generate a model card</h3><p>Another approach is to generate a model card, which is a free-form document that can be adjusted to external documentation requirements, such as those from a governance team within your own organization, or legal requirements. One example that we have come across is an organization that has inspected the source code of a few specific versions of a ML library, as a security measure. This organization therefore requires the ability to verify that any model that is approved for use in production, was trained with an approved version of the library.</p><p>Aside from the performance KPIs that are typically included in the evaluation as well, external requirements often demand “explainability”: Feature importance and performance across different data splits, such as different demographics, geographies, or, for instance, income ranges. This helps to spot any biases that your model may have, and it’s also a good place to look when trying to improve the model performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gmRzX0P1h0Lt5ekD" /><figcaption>Example of a part of a model card, generated using the <a href="https://www.tensorflow.org/responsible_ai/model_card_toolkit/guide">model card toolkit</a>.</figcaption></figure><h3>Upload model to a model registry</h3><p>Rather than leaving a model in a place that is convenient for the training system, we should store it somewhere where it can be properly cataloged, discovered and used. Such a place is often called a model registry. Important functions of a model registry are the storage of metadata, which includes the model name and other descriptive information, version control, and performance evaluation. The performance evaluation could include a precision-recall curve, a confusion matrix, etc, in case of a classifier.</p><p>Another aspect is that the model registry should contain all the required information to use the model for inference. Vertex AI Model Registry solves this by associating each model (version) with a container image for inference.</p><h3>Validate training data against a schema</h3><p>Validation of the training data is frequently done manually, by producing statistics of the different features (min, max, mean, median, etc), as well as histogramming the distributions. But for production pipelines, it’s important to produce a machine-readable file that describes our expectations for the data that is used for training, retraining, and inference. This way, we can be alerted when new data deviates from our expectation, and we can think about this before we train a new model with this data, and potentially even put it into production.</p><p>TFX implements this functionality out of the box through a SchemaGen component that will write the file, based on the training data. Like any automatic schema detection, this does not always work flawlessly, so one should edit the file by hand to prepare it for use. The schema file contains the data types of the features, but alsovalue dictionaries for categorical variables (by default, TFX will write the observed values from the training data), value ranges, and so on.</p><h3>Avoid training-serving skew due to data transformations</h3><p>Training-serving skew is not a term that is used consistently. The definition I use is “a difference between training and serving in the data or the treatment of the data”. That means it includes data drift, when the distribution of one or more features changes, compared to the training dataset. We should monitor for this effect, especially if we do not frequently retrain our model regardless of any data drift, but it’s not part of the training pipeline that we are concerned with here.</p><p>What does affect the training pipeline, is that the training pipeline often defines the transformations that create the features. For example, normalization of a variable involves subtracting the mean of the distribution and dividing by its standard deviation. This mean and standard deviation is calculated from the training dataset, so how do we apply this transformation during inference in exactly the same way? One way is to store the numbers we need somewhere, but it needs to be done in such a way that we always use the right ones. We may, for example, have multiple versions of the same model in production at the same time, in a canary deployment or A/B test. If the models are trained on different training data, then the transformations to be applied will be different.</p><p>Normalization is a simple example, but this problem can produce increasing levels of annoyance. Transformations that we need to replicate from training to inference in this way include one-hot encoding, embeddings, bucketing and more.</p><p>The way TFX solves this problem is by storing the transformations in a binary format during training and incorporating them into the model itself, before writing the model to disk.</p><p>If you’re not using TFX, but you are using a Tensorflow model, and your transformations are part of Tensorflow Transform, then you can <a href="https://www.tensorflow.org/tfx/tutorials/transform/simple#putting_it_all_together">do this yourself</a> too. If you’re not using Tensorflow, then the standard solution is to use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html">“pipeline” concept from your scikit-learn</a>. If your ML framework plays nice with scikit-learn, then you can build one pipeline of your transformation operations and your model together. This is a mini pipeline that concatenates the feature transformations and the model inferences and makes it look like a single operation. This is the case for <a href="https://goodboychan.github.io/python/datacamp/machine_learning/2020/07/07/03-Using-XGBoost-in-pipelines.html#Incorporating-XGBoost-into-pipelines">xgboost</a> and <a href="https://www.kaggle.com/code/paweljankiewicz/lightgbm-with-sklearn-pipelines">lightgbm</a>, for example, but not for PyTorch.</p><p>In the case of PyTorch, you could still encapsulate all your feature engineering operations into one scikit-learn pipeline, and then inference would require two steps: Running the feature engineering pipeline and passing the output to PyTorch. But scikit-learn pipelines do not scale to large datasets. If we need to run feature engineering on large datasets, then we will have to distribute the work, for example using Spark. Again, TFX has this problem solved for us already. TFX uses Tensorflow Transform which implements the transformations as an <a href="https://beam.apache.org/">Apache Beam pipeline</a>. These pipelines can be run “locally”, but they can also be trivially executed on other runners, such as on a Spark cluster. This means that the transformations defined using Tensorflow Transform are implemented as a Beam pipeline, which actually executes as a Spark job. Google Cloud provides a managed service to run Beam jobs called <a href="https://cloud.google.com/dataflow?hl=en">Dataflow</a>.</p><h3>Conclusions</h3><p>We’re sure we’ve missed many valuable best practices in this review, and even the ones we listed can be implemented more consistently in the reference pipelines that we provide. But we don’t want the perfect to be the enemy of the good: We hope this is a useful overview of our experience from the field. The <a href="https://github.com/GoogleCloudPlatform/professional-services/tree/main/examples/vertex_mlops_enterprise">source code</a> links to head versions of the code repository, and we will continue to make improvements.</p><p>Feel free to continue with our <a href="https://jgpuga.medium.com/enterprise-mlops-with-google-cloud-vertex-ai-part-3-ci-cd-33d5e6e774a7">last article in this MLOps series on CI/CD</a>, where we will discuss the automation of the development process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cba36819058" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/machine-learning-pipeline-development-on-google-cloud-5cba36819058">Machine Learning Pipeline Development on Google Cloud</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to calculate a correlation matrix on Big Data using Google BigQuery]]></title>
            <link>https://medium.com/google-cloud/how-to-calculate-a-correlation-matrix-on-big-data-using-google-bigquery-fb629fbf57a5?source=rss-94fa7ea880d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb629fbf57a5</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[correlation-coefficient]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Paul Balm]]></dc:creator>
            <pubDate>Sun, 22 Jan 2023 14:25:39 GMT</pubDate>
            <atom:updated>2023-01-23T14:53:25.527Z</atom:updated>
            <content:encoded><![CDATA[<p>BigQuery is Google Cloud’s serverless enterprise data warehouse. It’s serverless, offers scalable analytics, streaming ingestion, and machine learning right in the database. (I like the <a href="https://www.linkedin.com/posts/elena-mata-yandiola_very-pleased-to-have-spoken-at-the-cloud-activity-7004383043528372224-E_-5/?utm_source=share&amp;utm_medium=member_desktop">ML capabilities</a> a lot. :-) ) But I’m guessing if you found this post, you already know about BigQuery.</p><p>You can use BigQuery to calculate the correlation coefficient between two columns using the <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions#corr">CORR() function</a>, as Google already explained in 2013 in <a href="https://cloudplatform.googleblog.com/2013/09/introducing-corr-to-google-bigquery.html">a detailed post</a>. This calculation scales to virtually any table size, which is great, but what if you wanted to calculate the full correlation matrix?</p><figure><img alt="Some colorful threads woven together in a square pattern. Imaged generated by DALL-E." src="https://cdn-images-1.medium.com/max/1024/1*L9FCgzAnbbn7BZ1OUSKDaw.png" /></figure><h3>A naive solution</h3><p>We can just <strong>calculate all the correlation coefficients</strong>. If we have three columns, a, b, and c, in a table t, we can just calculate the correlation coefficients running queries like <strong><em>SELECT CORR(a, b) FROM t</em>, <em>SELECT CORR(a, c) FROM t</em>, an so on.</strong> Depending on how many columns you have, you will probably get tired of typing these repetitive queries. Good news! BigQuery allows programmatic (SQL) access to the columns of your table, in fact the entire schema, through the <a href="https://cloud.google.com/bigquery/docs/information-schema-columns"><em>COLUMNS</em> view in the <em>INFORMATION_SCHEMA</em> dataset</a>. Let’s have a look at an example using the bike share plan data from the City of London that BigQuery makes available to all users through the <a href="https://cloud.google.com/bigquery/public-data">BigQuery public datasets</a>.</p><p>In a Jupyter Notebook (perhaps a <a href="https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction">serverless managed notebook on Vertex AI Workbench</a>), you can use the <a href="https://cloud.google.com/python/docs/reference/bigquery/latest/magics"><em>bigquery</em> cell-magic</a> to send queries to BigQuery and parse the output directly into a Pandas Dataframe:</p><pre>%%bigquery df_cols<br><br>SELECT column_name<br>FROM `bigquery-public-data.london_bicycles.INFORMATION_SCHEMA.COLUMNS`<br>WHERE table_name = “cycle_hire” AND data_type = “INT64”</pre><p>Note that I’m selecting only the numerical columns, because the correlation coefficient can only be calculated between numerical columns. All numerical columns in this particular table happen to be of integer type, 8 of them.</p><p>We now have a Dataframe called “df_cols” with our columns, which allows us to generate the queries:</p><pre>queries = []<br>table_name = “bigquery-public-data.london_bicycles.cycle_hire”<br>for i in range(len(cols)):<br>  for j in range(i+1, len(cols)):<br>    queries.append(f”SELECT CORR({cols[i]}, {cols[j]}) as corr_{i}_{j} FROM `{table_name}`”)</pre><p>We can now run these queries to calculate correlation coefficients, from our notebook, using the <a href="https://cloud.google.com/python/docs/reference/bigquery/latest">BigQuery Python SDK</a>, for example for the first query</p><pre>%%time <br><br>from google.cloud.bigquery import Client<br><br>client = Client()<br><br>job = client.query(queries[0])<br>df_corr = job.to_dataframe()<br><br>coefficient = df_corr[&#39;corr_0_1&#39;][0]</pre><p>I have used the <em>%%time</em> cell magic here, which calculates the time it takes to run this cell. I just ran it, it took more than 3 seconds… I don’t have to rerun the import or recreate the client for every query, but BigQuery is optimized for high-volume analytical queries, not for low latency.</p><p>An 8 by 8 correlation matrix requires the calculation of 28 correlation coefficients. This would take perhaps 1–2 minutes, which is still manageable, but it gets out of hand quadratically as the number of columns grows: For 16 columns (twice as many) it would take 4 times as long, and if a table has 64 numerical columns you can expect to have to wait perhaps half an hour.</p><p>There are solutions to reduce the latency of queries on BigQuery, such as <a href="https://cloud.google.com/bigquery/docs/bi-engine-intro">BigQuery BI Engine</a>, but I doubt it would work in this case where all queries are different. In any case, I have a better idea.</p><h3>Combining all correlation coefficients in a single query</h3><p>If running many queries on BigQuery takes time, then why don’t we just combine everything into one query?</p><pre>corr_calls = []<br>table_name = &quot;bigquery-public-data.london_bicycles.cycle_hire&quot;<br>for i in range(len(cols)):<br>    for j in range(i+1, len(cols)): <br>        corr_calls.append(f&quot;CORR({cols[i]}, {cols[j]}) as corr_{i}_{j}&quot;)<br>        <br>one_query = f&quot;SELECT {&#39;, &#39;.join(corr_calls)} FROM `{table_name}`&quot;</pre><p>Now the query looks like this:</p><blockquote>SELECT CORR(rental_id, duration) as corr_0_1, CORR(rental_id, bike_id) as corr_0_2, CORR(rental_id, end_station_id) as corr_0_3, CORR(rental_id, start_station_id) as corr_0_4, CORR(rental_id, end_station_logical_terminal) as corr_0_5, CORR(rental_id, start_station_logical_terminal) as corr_0_6, CORR(rental_id, end_station_priority_id) as corr_0_7, CORR(duration, bike_id) as corr_1_2, CORR(duration, end_station_id) as corr_1_3, CORR(duration, start_station_id) as corr_1_4, CORR(duration, end_station_logical_terminal) as corr_1_5, CORR(duration, start_station_logical_terminal) as corr_1_6, CORR(duration, end_station_priority_id) as corr_1_7, CORR(bike_id, end_station_id) as corr_2_3, CORR(bike_id, start_station_id) as corr_2_4, CORR(bike_id, end_station_logical_terminal) as corr_2_5, CORR(bike_id, start_station_logical_terminal) as corr_2_6, CORR(bike_id, end_station_priority_id) as corr_2_7, CORR(end_station_id, start_station_id) as corr_3_4, CORR(end_station_id, end_station_logical_terminal) as corr_3_5, CORR(end_station_id, start_station_logical_terminal) as corr_3_6, CORR(end_station_id, end_station_priority_id) as corr_3_7, CORR(start_station_id, end_station_logical_terminal) as corr_4_5, CORR(start_station_id, start_station_logical_terminal) as corr_4_6, CORR(start_station_id, end_station_priority_id) as corr_4_7, CORR(end_station_logical_terminal, start_station_logical_terminal) as corr_5_6, CORR(end_station_logical_terminal, end_station_priority_id) as corr_5_7, CORR(start_station_logical_terminal, end_station_priority_id) as corr_6_7 FROM `bigquery-public-data.london_bicycles.cycle_hire`</blockquote><p>Ah! The smell of generated code in the morning.</p><p>In the BigQuery UI, this query processes about 1 GB of data and returns pretty much immediately. It reports “0 seconds” execution time, even after disabling the <a href="https://cloud.google.com/bigquery/docs/cached-results">BigQuery cache</a>.</p><p>That looks great! Let’s try this with a bit wider table. The US Census Bureau publishes the American Community Survey (ACS), and it’s available in BigQuery as public data as well, for you to explore. It has 244 numeric columns. Now our query looks like this:</p><figure><img alt="Zoomed out, unreadable screenshot of a very very long query." src="https://cdn-images-1.medium.com/max/1024/1*yAiykF8iq7_PGSR8m5AV8w.png" /></figure><p>That’s a bit zoomed out, and it’s still only roughly the first 10% of the query. I guess… We’re going to send this query, all 1,997,998 characters of it, to BigQuery. Unfortunately, that won’t work. BigQuery responds: “The query is too large. The maximum standard SQL query length is 1,024,000 characters, including comments and white space characters.” We’re less than a factor two over the limit. Not even that much!</p><p>It could be that you don’t run into this limitation if you try this on your data, even if you have a similar number of numerical columns. The column names are quite long in this dataset, like “aggregate_travel_time_to_work”. You can also try to work around it with an initial query that shortens the columns names, like <em>SELECT aggregate_travel_time_to_work as col_17 …</em> followed by <em>SELECT CORR(col_17, …</em> — that would probably reduce the length of the query string.</p><p>Perhaps we could try with the first… 150 columns or something. That gives us a query under a million characters. What does BigQuery say? Oops: “Too many total leaf fields: 11,175. Max allowed field count: 10,000”. Also pretty close! But as documented on the <a href="https://cloud.google.com/bigquery/quotas">BigQuery quotas and limits</a> page, “Each table, query result, or view definition can have up to 10,000 columns”. That translates to just over 140 columns. The correlation matrix for such a table is actually 140x140, but we only calculate about half of it, since the other half is the same.</p><p>Do we have to go back to the original idea, and run 11,175 individual queries, which would take several hours at least?</p><h3>A Combined Approach That Scales</h3><p>Rather than going back all the way to one query per correlation coefficient, we can break up our queries to generate a list of queries, each of which remains below the limit. In the case of the ACS table with 244 columns, the full correlation matrix has just under 60,000 entries, so we can break this into 6 queries, calculating each just under 10,000 coefficients. Then we combine the results and we build the full matrix. Compared to our naive approach which lasts several hours, this gives a great improvement, because it runs in about 3 minutes.</p><p>If we run these queries like we have run them before, they run at the highest priority, as “interactive queries”. The BigQuery Python SDK does not allow you to run them in parallel, though. For that, you have to <a href="https://cloud.google.com/bigquery/docs/running-queries#python">run them as “batch queries”</a>, and you will be allowed 10 queries in parallel and up to 20,000 queries in the the queue. Running this way, <strong>we can calculate all 59,536 correlation coefficients from 244 columns with each 85,395 rows in just 1 minute</strong>. It’s done as follows.</p><p>First we query the <em>COLUMNS</em> view in the <em>INFORMATION_SCHEMA</em> to get the list of columns, and we generate all the calls to the <em>CORR</em> function: <em>CORR(col_1, col_2), CORR(col_1, col_3)</em>, and so on:</p><pre>from google.cloud.bigquery import Client, QueryJobConfig, QueryPriority<br>import numpy as np<br>import matplotlib<br>import matplotlib as mpl<br>import matplotlib.pyplot as plt<br>import os<br>import pandas as pd<br><br><br>project = &#39;bigquery-public-data&#39;<br>dataset = &#39;census_bureau_acs&#39;<br>table = &#39;censustract_2020_5yr&#39;<br><br><br>query = f&quot;&quot;&quot;<br>    SELECT column_name, data_type <br>    FROM {project}.{dataset}.INFORMATION_SCHEMA.COLUMNS <br>    WHERE table_name = &#39;{table}&#39;<br>&quot;&quot;&quot;<br><br>client = Client()<br>job = client.query(query)<br>df_cols = job.to_dataframe()<br><br>cols_clean = []<br>cols_for_corr = []<br>for row in df_cols.iterrows():<br>    col = row[1][0]<br>    col_clean = col<br>    dtype = row[1][1]<br>    <br>    # if it&#39;s an INT64, cast it to FLOAT64 for CORR calculation<br>    if dtype == &#39;INT64&#39;:<br>        col = f&#39;CAST({col} AS FLOAT64)&#39;<br>    <br>    if dtype in [&#39;INT64&#39;, &#39;FLOAT64&#39;]:<br>        cols_for_corr.append(col)<br>        cols_clean.append(col_clean)<br><br>    corr_calls = []<br>    for (n1, col1) in enumerate(cols_for_corr):<br>        for (n2, col2) in enumerate(cols_for_corr):<br>            # CORR(col1, col2) as c_1_2<br>            corr_calls.append(f&#39;CORR({col1}, {col2}) AS c_{n1}_{n2}&#39;)</pre><p>Now we combine the calls to the CORR function into queries, with each a bit less than 10,000 elements. We then launch these queries in batch mode:</p><pre>max_coeffs_per_query = 9991<br>#max_coeffs_per_query = 90<br><br>i = 0<br><br>jobs = []<br><br>job_config = QueryJobConfig(<br>    # Run at batch priority, which won&#39;t count toward concurrent rate limit.<br>    priority=QueryPriority.BATCH,<br>    use_query_cache=False # to ensure we&#39;re measuring the time correctly<br>)<br><br>while i &lt; len(corr_calls):<br>    print(i)<br><br>    filename = f&#39;data_batch/df_{i}&#39;<br>    if not os.path.isfile(&#39;file&#39;):<br>        # combine calls to CORR in SQL statement<br>        query = f&quot;SELECT {&#39;, &#39;.join(corr_calls[i:i + max_coeffs_per_query])} FROM {project}.{dataset}.{table}&quot;<br>        job = client.query(query, job_config=job_config)<br>        jobs.append(job)<br>    i = i + max_coeffs_per_query <br><br>print(f&#39;Running {len(jobs)} BigQuery jobs in parallel&#39;)</pre><p>For this case, there will be 6 jobs in parallel, assuming there are no other batch jobs in the project running at the time, taking up more than 4 of 10 available slots for running batch jobs simultaneously.</p><p>Now we just have to wait for the jobs to complete:</p><pre>for job in jobs:<br>    while not job.done():<br>        time.sleep(1)</pre><p>In order to avoid having to run the queries again, let’s extract the data and save it locally. It’s about 1 MB. Create a directory “data_batch” first.</p><pre>for i, job in enumerate(jobs):<br>        df_corrs = job.to_dataframe()<br>        filename = f&#39;data_batch/df_{i}&#39;<br>        df_corrs.to_csv(filename)<br>        print(f&quot;  {filename}&quot;)</pre><p>Now we read the data back in, and combine the results of the queries in a single Dataframe:</p><pre>df_corrs = None<br><br>for i in range(len(jobs)):<br>    print(i)<br><br>    filename = f&#39;data_batch/df_{i}&#39;<br>    df_i = pd.read_csv(filename)<br>    if df_corrs is None:<br>        df_corrs = df_i<br>    else:<br>        df_corrs = pd.concat([df_corrs, df_i], axis=1, ignore_index=False)<br>    i = i + max_coeffs_per_query</pre><p>Reading the data back and combining it like this, leads to one additional column per file that corresponds to the index of the Dataframe for that file. We don’t need those columns, let’s drop them.</p><pre>for col in df_corrs.columns:<br>    if col.startswith(&quot;Unnamed&quot;) and col in df_corrs.columns:<br>        print(col)<br>        df_corrs.drop(columns=[col], inplace=True)</pre><p>And that’s it! We have all the data for the correlation matrix:</p><figure><img alt="Screenshot of the dataframe with all the correlation coefficients. It shows one row with each coefficient in its own column." src="https://cdn-images-1.medium.com/max/1024/1*ktlZuQDaD4zQ1pwnjiKvqg.png" /></figure><p>But this is not the most friendly representation. I’ll create a <a href="https://matplotlib.org/stable/gallery/images_contours_and_fields/image_annotated_heatmap.html">heatmap</a> to show what the data looks like. The full code is in my notebook on Github. The result is a nice image!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*jEsBBNiAzoUotH88eaupuQ.png" /></figure><p>I’ll include the legend for the column numbers below this article.</p><h3>A final speed-up</h3><p>In order to get an aesthetically pleasing result, I have calculate all elements of the correlation matrix, essentially duplicating the work on BigQuery. We can speed up even more by calculating only half, and copying the data into the lower half of the correlation matrix.</p><p>We can just eliminate the duplicate calls to the <em>CORR</em> function (eliminate <em>CORR(b,a)</em> if we have already calculated <em>CORR(a,b)</em>). This reduces the number of elements that we are calculating, but we’re not speeding up that much. If each query still calculates 10,000 elements, then there are now 3 of them, rather than 6. But each still takes the same time as before, and we need to wait for them to finish. So we can tune the code, so that each query calculates 3,000 correlation coefficients. That way, we run 10 jobs in parallel, filling exactly the 10 available batch job slots.</p><p>It turns out that the slowest part is the extraction of the data from each job, which we still do sequentially. But even so, this optimization brings us down to <strong>about 40 seconds to calculate and extract all data for this correlation matrix</strong>.</p><h3>References</h3><p>A <strong>notebook with the code</strong> for this article <a href="https://github.com/pbalm/bigquery_samples/blob/main/bigquery_correlation_matrix.ipynb">can be found on Github</a>.</p><p>BigQuery:</p><ul><li><a href="https://cloud.google.com/bigquery">BigQuery product overview</a></li><li><a href="https://www.linkedin.com/posts/elena-mata-yandiola_very-pleased-to-have-spoken-at-the-cloud-activity-7004383043528372224-E_-5/?utm_source=share&amp;utm_medium=member_desktop">My presentation on BigQuery ML with the great Elena Mata Yandiola</a> during the Digital Natives Cloud Developer Day in Berlin in late 2022</li><li><a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/statistical_aggregate_functions">BigQuery statistical functions</a> and a <a href="https://cloudplatform.googleblog.com/2013/09/introducing-corr-to-google-bigquery.html">post about the introduction of the CORR function</a> in 2013</li><li>The <a href="https://cloud.google.com/bigquery/docs/information-schema-columns">BigQuery COLUMNS view</a>, to programmatically (using SQL) access the schema of your tables</li><li><a href="https://cloud.google.com/bigquery/public-data">BigQuery public datasets</a></li><li><a href="https://cloud.google.com/bigquery/quotas">BigQuery quotas and limits</a></li><li><a href="https://cloud.google.com/python/docs/reference/bigquery/latest">BigQuery Python SDK</a> and <a href="https://cloud.google.com/bigquery/docs/running-queries#python">how to run interactive and batch queries</a></li><li><a href="https://cloud.google.com/python/docs/reference/bigquery/latest/magics">BigQuery magics</a> for Jupyter notebooks</li><li><a href="https://cloud.google.com/bigquery/docs/cached-results">Using cached query results</a></li><li><a href="https://cloud.google.com/bigquery/docs/bi-engine-intro">BI Engine</a>: “A fast, in-memory analysis service that accelerates many SQL queries in BigQuery by intelligently caching the data you use most frequently”</li></ul><p>Vertex AI Platform:</p><ul><li><a href="https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction">Managed notebooks</a>: Serverless Jupyter notebooks</li></ul><h3>Legend for the heatmap</h3><pre>0: aggregate_travel_time_to_work<br>1: amerindian_including_hispanic<br>2: amerindian_pop<br>3: armed_forces<br>4: asian_including_hispanic<br>5: asian_male_45_54<br>6: asian_male_55_64<br>7: asian_pop<br>8: associates_degree<br>9: bachelors_degree<br>10: bachelors_degree_2<br>11: bachelors_degree_or_higher_25_64<br>12: black_including_hispanic<br>13: black_male_45_54<br>14: black_male_55_64<br>15: black_pop<br>16: children<br>17: children_in_single_female_hh<br>18: civilian_labor_force<br>19: commute_10_14_mins<br>20: commute_15_19_mins<br>21: commute_20_24_mins<br>22: commute_25_29_mins<br>23: commute_30_34_mins<br>24: commute_35_39_mins<br>25: commute_35_44_mins<br>26: commute_40_44_mins<br>27: commute_45_59_mins<br>28: commute_5_9_mins<br>29: commute_60_89_mins<br>30: commute_60_more_mins<br>31: commute_90_more_mins<br>32: commute_less_10_mins<br>33: commuters_16_over<br>34: commuters_by_bus<br>35: commuters_by_car_truck_van<br>36: commuters_by_carpool<br>37: commuters_by_public_transportation<br>38: commuters_by_subway_or_elevated<br>39: commuters_drove_alone<br>40: different_house_year_ago_different_city<br>41: different_house_year_ago_same_city<br>42: dwellings_10_to_19_units<br>43: dwellings_1_units_attached<br>44: dwellings_1_units_detached<br>45: dwellings_20_to_49_units<br>46: dwellings_2_units<br>47: dwellings_3_to_4_units<br>48: dwellings_50_or_more_units<br>49: dwellings_5_to_9_units<br>50: employed_agriculture_forestry_fishing_hunting_mining<br>51: employed_arts_entertainment_recreation_accommodation_food<br>52: employed_construction<br>53: employed_education_health_social<br>54: employed_finance_insurance_real_estate<br>55: employed_information<br>56: employed_manufacturing<br>57: employed_other_services_not_public_admin<br>58: employed_pop<br>59: employed_public_administration<br>60: employed_retail_trade<br>61: employed_science_management_admin_waste<br>62: employed_transportation_warehousing_utilities<br>63: employed_wholesale_trade<br>64: families_with_young_children<br>65: family_households<br>66: father_in_labor_force_one_parent_families_with_young_children<br>67: father_one_parent_families_with_young_children<br>68: female_10_to_14<br>69: female_15_to_17<br>70: female_18_to_19<br>71: female_20<br>72: female_21<br>73: female_22_to_24<br>74: female_25_to_29<br>75: female_30_to_34<br>76: female_35_to_39<br>77: female_40_to_44<br>78: female_45_to_49<br>79: female_50_to_54<br>80: female_55_to_59<br>81: female_5_to_9<br>82: female_60_to_61<br>83: female_62_to_64<br>84: female_65_to_66<br>85: female_67_to_69<br>86: female_70_to_74<br>87: female_75_to_79<br>88: female_80_to_84<br>89: female_85_and_over<br>90: female_female_households<br>91: female_pop<br>92: female_under_5<br>93: four_more_cars<br>94: gini_index<br>95: graduate_professional_degree<br>96: group_quarters<br>97: high_school_diploma<br>98: high_school_including_ged<br>99: hispanic_any_race<br>100: hispanic_male_45_54<br>101: hispanic_male_55_64<br>102: hispanic_pop<br>103: households<br>104: households_public_asst_or_food_stamps<br>105: households_retirement_income<br>106: housing_built_1939_or_earlier<br>107: housing_built_2000_to_2004<br>108: housing_built_2005_or_later<br>109: housing_units<br>110: housing_units_renter_occupied<br>111: in_grades_1_to_4<br>112: in_grades_5_to_8<br>113: in_grades_9_to_12<br>114: in_school<br>115: in_undergrad_college<br>116: income_100000_124999<br>117: income_10000_14999<br>118: income_125000_149999<br>119: income_150000_199999<br>120: income_15000_19999<br>121: income_200000_or_more<br>122: income_20000_24999<br>123: income_25000_29999<br>124: income_30000_34999<br>125: income_35000_39999<br>126: income_40000_44999<br>127: income_45000_49999<br>128: income_50000_59999<br>129: income_60000_74999<br>130: income_75000_99999<br>131: income_less_10000<br>132: income_per_capita<br>133: less_one_year_college<br>134: less_than_high_school_graduate<br>135: male_10_to_14<br>136: male_15_to_17<br>137: male_18_to_19<br>138: male_20<br>139: male_21<br>140: male_22_to_24<br>141: male_25_to_29<br>142: male_30_to_34<br>143: male_35_to_39<br>144: male_40_to_44<br>145: male_45_64_associates_degree<br>146: male_45_64_bachelors_degree<br>147: male_45_64_grade_9_12<br>148: male_45_64_graduate_degree<br>149: male_45_64_high_school<br>150: male_45_64_less_than_9_grade<br>151: male_45_64_some_college<br>152: male_45_to_49<br>153: male_45_to_64<br>154: male_50_to_54<br>155: male_55_to_59<br>156: male_5_to_9<br>157: male_60_to_61<br>158: male_62_to_64<br>159: male_65_to_66<br>160: male_67_to_69<br>161: male_70_to_74<br>162: male_75_to_79<br>163: male_80_to_84<br>164: male_85_and_over<br>165: male_male_households<br>166: male_pop<br>167: male_under_5<br>168: management_business_sci_arts_employed<br>169: married_households<br>170: masters_degree<br>171: median_age<br>172: median_income<br>173: median_rent<br>174: median_year_structure_built<br>175: million_dollar_housing_units<br>176: mobile_homes<br>177: mortgaged_housing_units<br>178: no_car<br>179: no_cars<br>180: nonfamily_households<br>181: not_hispanic_pop<br>182: not_in_labor_force<br>183: not_us_citizen_pop<br>184: occupation_management_arts<br>185: occupation_natural_resources_construction_maintenance<br>186: occupation_production_transportation_material<br>187: occupation_sales_office<br>188: occupation_services<br>189: occupied_housing_units<br>190: one_car<br>191: one_parent_families_with_young_children<br>192: one_year_more_college<br>193: other_race_pop<br>194: owner_occupied_housing_units<br>195: owner_occupied_housing_units_lower_value_quartile<br>196: owner_occupied_housing_units_median_value<br>197: owner_occupied_housing_units_upper_value_quartile<br>198: percent_income_spent_on_rent<br>199: pop_16_over<br>200: pop_25_64<br>201: pop_25_years_over<br>202: pop_5_years_over<br>203: pop_determined_poverty_status<br>204: pop_in_labor_force<br>205: population_1_year_and_over<br>206: population_3_years_over<br>207: poverty<br>208: rent_10_to_15_percent<br>209: rent_15_to_20_percent<br>210: rent_20_to_25_percent<br>211: rent_25_to_30_percent<br>212: rent_30_to_35_percent<br>213: rent_35_to_40_percent<br>214: rent_40_to_50_percent<br>215: rent_burden_not_computed<br>216: rent_over_50_percent<br>217: rent_under_10_percent<br>218: renter_occupied_housing_units_paying_cash_median_gross_rent<br>219: sales_office_employed<br>220: some_college_and_associates_degree<br>221: speak_only_english_at_home<br>222: speak_spanish_at_home<br>223: speak_spanish_at_home_low_english<br>224: three_cars<br>225: total_pop<br>226: two_cars<br>227: two_or_more_races_pop<br>228: two_parent_families_with_young_children<br>229: two_parents_father_in_labor_force_families_with_young_children<br>230: two_parents_in_labor_force_families_with_young_children<br>231: two_parents_mother_in_labor_force_families_with_young_children<br>232: two_parents_not_in_labor_force_families_with_young_children<br>233: unemployed_pop<br>234: vacant_housing_units<br>235: vacant_housing_units_for_rent<br>236: vacant_housing_units_for_sale<br>237: walked_to_work<br>238: white_including_hispanic<br>239: white_male_45_54<br>240: white_male_55_64<br>241: white_pop<br>242: worked_at_home<br>243: workers_16_and_over</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb629fbf57a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/how-to-calculate-a-correlation-matrix-on-big-data-using-google-bigquery-fb629fbf57a5">How to calculate a correlation matrix on Big Data using Google BigQuery</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[De-identifying data using custom info types with DLP on Google Cloud]]></title>
            <link>https://medium.com/google-cloud/de-identifying-data-using-custom-info-types-with-dlp-on-google-cloud-122f5729b0f9?source=rss-94fa7ea880d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/122f5729b0f9</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[dlp]]></category>
            <dc:creator><![CDATA[Paul Balm]]></dc:creator>
            <pubDate>Sat, 01 Oct 2022 13:44:56 GMT</pubDate>
            <atom:updated>2022-10-02T02:15:15.091Z</atom:updated>
            <content:encoded><![CDATA[<p>Google Cloud offers a service to find and de-identify sensitive data, through masking, encryption, or other transformations. This service is called Data Loss Prevention, or DLP. At the time of writing, it has 158 types of sensitive data, or “info types”, predefined. These range from simple things like genders, to Spanish national identity numbers, Japanese bank account numbers, and US vehicle identification numbers. But what if you have a list of items that are not covered by any of these? Like the schools in your country, or perhaps professions, or something else… A dictionary of a few words, or a massive one, or a regular expression maybe? You need a custom info type. This post will describe how that works, but first, the basics.</p><h3>Data Loss Prevention Basics</h3><p>A DLP job will always have an “inspection” part, with its corresponding “inspection config”. And optionally, it can do something with the findings from the inspection, such as encrypting it. This is done in the “de-identification”.</p><p>The inspection config will specify the info types to be identified, and the de-identification will define the actions to be applied where necessary. For example, we could identify bank account numbers and encrypt them (reversible), or hash them (not reversible), or mask them by overwriting them with a certain character like “X”. There are many other options too, for example, dates can all be shifted by some amount of time, to preserve the timeline of events in the dataset.</p><p>In the diagram below we show three predefined info types. The inspection job only looks for two of them, the ID number and the vehicle ID. Reducing the number of info types reduces the latency of the inspection jobs. In the de-identification step, we only apply a transformation to the ID number, for example, we could hash it. We can also define inspection and de-identification templates to ease the management of the configuration of different job, but this does not change the concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yZQkt0coss8NTe1JO4hx2Q.png" /><figcaption>Info types can be used by an inspection job. During the optional de-identification step, transformations can be applied to findings based on some of the info types from the inspection job.</figcaption></figure><p>Predefined info types, like these, have an ID, which we can see, and it has an implementation, that is not exposed. We do not know the algorithm to recognise Japanese bank account numbers and we can’t find out. We simply refer to the predefined info type by its ID.</p><h3>Introducing Custom Info Types</h3><p>If we have a need to that none of the predefined info types can meet, we may need to define an info type ourselves. There are a number of different custom types, but basically there is one based on regular expressions and there are a couple more, based on dictionaries. The dictionary can just be a list of words that we enter into the info type itself, or it can be a text file on Google Cloud Storage, or if we have a very large dictionary, it can be stored on BigQuery.</p><p>Note how we are discussing here how the different info types are implemented. This is an important difference with the predefined info types.</p><p>Custom info types can be defined as we create the inspection job, or we can store the custom info type definition separately as a “stored info type”, so that we can reuse it in another inspection job. Let’s say we want to mask information in our data about the schools that people have attended and the level at which they have studied, which can only be “high school” or “college”, for simplicity’s sake. The number of schools in the country is a big dictionary though, that we may want to reuse. So let’s create a stored info type for all the schools, and we can define the study level within the inspection job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pEBHEiynaWm7aDMAUsRIWw.png" /><figcaption>A mix of predefined and custom info types (marked by a star), that is used in an inspection job and a de-identification process.</figcaption></figure><p>In the inspection job, we have first added a custom info type for the level of studies, based on a word list of just two words, and we have not stored it externally.</p><p>The info type to identify schools is stored outside of the inspection config. The stored info type is assigned a name for easy identification, it gets a fully qualified resource ID, and it has an implementation: In this case, a dictionary on BigQuery. When we include the school info type in our inspect config, we are free to give it the name that we want. I have chosen “school-name” here, to distinguish from the stored info type name “school”. The inspect config refers back to the stored type through its fully qualified resource ID, not by its name.</p><p>The custom info types in the inspect config have another attribute, that doesn’t appear in the diagram: A likelihood. Findings based on predefined info types are assigned a likelihood depending on, for example, how likely a series of numbers refers to a Japanese bank account. For the stored info types, we can specify the probability that our findings are assigned, ranging from “very unlikely” to “very likely”. At the level of the job we can then specify that we only want to see the findings that are at least “likely”, for example. If we set the findings from our custom info type to “unlikely”, they will be filtered. This kind of tweaking can reduce noise, but you can also save all findings to BigQuery, and do any filtering there.</p><p>These are the concepts, let’s see how this is done in the Google Cloud Console!</p><h3>Demo Time</h3><p>We’re going to create a stored info type that detects school names based on a word list, define an inspection template using this info type, and a de-identification template to redact the schools from text.</p><p>First, we’ll hop over to the <a href="https://pantheon.corp.google.com/security/dlp/landing/configuration/infoTypes/stored">stored info types section of the DLP configuration</a> for your project. Hit the “Create Stored InfoType” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*od9WwGFFUEfnWjAyO36a_g.png" /><figcaption>The stored info types section of the DLP Configuration in the Google Cloud Console.</figcaption></figure><p>We said we’re going to create a stored info type on a word list, which in this demo, saves us the trouble of having to create an external dictionary. So select “List of words or phrases” for the type and enter “school” for the InfoType ID. We’ll call this info type “school” and the list of schools that we’ll recognise is going to be limited to “harvard”, “yale”, and “brown”. Note that this will recognise “Harvard” regardless of which letters are capitals and which are not. Create the info type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*6_3VdAiBh5o_-vADvn32PQ.png" /><figcaption>Creating a stored infotype based on a word list.</figcaption></figure><p>If you go back to the stored infotypes section of the DLP configuration, you should now see an infotype called “school” there. You may want to click on it and copy the ID of the infotype, because we’ll need it later. It will be something like “projects/YOUR-PROJECT-ID/locations/global/storedInfoTypes/school”, with your project ID inserted at the right location of course.</p><p>Now let’s create an inspection template, that we can later use to quickly create inpection jobs to look for school names. In the <a href="https://pantheon.corp.google.com/security/dlp/landing/configuration/templates/inspect">inspection templates section of the DLP configuration</a>, click “create template”.</p><p>Under step 1 of the template creation process, we’ll just set the template ID to “find-schools”. Under step 2, we click “manage infotypes” and in the side panel that opens, we choose “custom”. There are no types listed there, so let’s add our stored infotype to this template. Click “Add custom infotype”, select “Stored InfoType” for the type, use “school-name” to name it, and provide the ID of the stored infotype that we created earlier (remember to replace “YOUR-PROJECT_ID” with you actual project ID).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_aToPoxDkurwGETyhIX0Q.png" /><figcaption>Adding our custom infotype to the inspection template.</figcaption></figure><p>Save the infotype and create the template. You should now see your inspection template configuration, as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*6pwsjEwsYn0nq7qah6LgZg.png" /><figcaption>The “find-schools” inspection template with a custom stored infotype.</figcaption></figure><p>Now finally, we’re going to create a de-identification template, and test it, to see if it works. Head over to the <a href="https://pantheon.corp.google.com/security/dlp/landing/configuration/templates/deidentify">de-identification template section of the DLP configuration</a> and hit “Create Template” there.</p><p>In the “Define template”, we’ll use “redact-schools” as the template ID. Under step 2, “Configure de-identification” we’ll choose “replace with infotype name” as the method. We could just apply this rule to all the infotypes from the inspection template, but let’s make this explicit. Select “specify infotypes”, and “manage infotypes”. A side panel opens where we can select any of the built-in infotypes, or we can select “name-only”. Click that, and “add infotype name”. Here, we have to add the name that we gave this infotype in the inspection template, “school-name”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tpZwhV5hWydpmxBHmEdjqw.png" /><figcaption>Adding the custom infotype to the de-identification template.</figcaption></figure><p>Click “done” and make sure that the “school-name” infotype is now listed. Click “create” to create the template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*whgvCnvSDil2uzCDHb0fgA.png" /><figcaption>Details of the de-identification template.</figcaption></figure><p>Now we finally arrive at the hour of truth. Hit “test”, and see if this works. Note that the “test” page specifies that any text that you enter, will be scanned with a default list of info types. Logically, our new custom info type is not part of this list. So we will have to provide our inspection template for this to work. Enter some text to see if it works!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*tW8trKI3CAyDtAE1QjP15A.png" /><figcaption>Testing the de-identification template.</figcaption></figure><p>Using custom info types gives you great freedom to scan for and de-identify very domain specific information, such a local school names, but also departments or using regular expressions, typical identifiers for certain use cases, such as ticket numbers or booking references from the airline industry.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=122f5729b0f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/de-identifying-data-using-custom-info-types-with-dlp-on-google-cloud-122f5729b0f9">De-identifying data using custom info types with DLP on Google Cloud</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Houston, we have a problem: Six Apollo Mission Principles for Pipeline Design]]></title>
            <link>https://medium.com/google-cloud/houston-we-have-a-problem-six-apollo-mission-principles-for-pipeline-design-20f3d0e0760c?source=rss-94fa7ea880d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/20f3d0e0760c</guid>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Paul Balm]]></dc:creator>
            <pubDate>Tue, 06 Sep 2022 16:47:01 GMT</pubDate>
            <atom:updated>2022-09-09T10:47:27.954Z</atom:updated>
            <content:encoded><![CDATA[<p>The Apollo 11 mission landed on the moon on July the 20th, 1969. In spite of the common narrative that Commander Armstrong bravely seized manual control of the spacecraft during the landing, the truth is that the Apollo Guidance Computer was in control all along, and functioning perfectly.</p><p>The Apollo Guidance computer was able to guide the spacecraft through complex procedures in an unpredictable environment and land two astronauts on the moon. It was able to do this, because it was designed according to 6 principles that permit the operation of complex calculations, not on our laptop but on a worker that we can’t directly access, in an unpredictable environment. And it was able to do this very, very reliably.</p><p>The Apollo Guidance Computer has properties that are desirable for our data processing pipelines today as well, and we could get closer to our ideals by applying these same design principles that got Buzz Aldrin and Neil Armstrong onto the surface of the moon, and perhaps even more impressively, safely back to Earth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*tdaVDA2xZPoAsveA" /><figcaption>The Apache Beam mascot</figcaption></figure><p>This story is told at its best by Robert Willis, in his entertaining talk titled “Light Years Ahead — The 1969 Apollo Guidance Computer”.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FB1J2RMorJXM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DB1J2RMorJXM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FB1J2RMorJXM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3634582797aa729cd42a78f5d21b8ff/href">https://medium.com/media/c3634582797aa729cd42a78f5d21b8ff/href</a></iframe><p>In this post, we will apply the principles from his presentation to modern data processing pipeline design, using <a href="https://beam.apache.org/">Apache Beam</a>. If you prefer video, we have presented this information at the Beam Summit 2022.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdGREGVW27_0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdGREGVW27_0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdGREGVW27_0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbdc44650720e86ec2a6211373b05ccc/href">https://medium.com/media/dbdc44650720e86ec2a6211373b05ccc/href</a></iframe><h3>What is a data processing pipeline?</h3><p>Running a data processing pipeline in the cloud is like launching into space. There is limited control after launch, you can only observe what happens, and hope for the best. If you made a mistake in your pipeline, the impact can be important. The operational cost of compromised reliability, resilience, maintainability and ease of operation can easily overshadow the development cost.</p><p>Like before a launch of a spacecraft, we need to prepare for the launch of our pipeline by applying some of the same design principles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*uV09H_0p65zNJwFwRclXMA.png" /><figcaption>The interface of a data processing pipeline</figcaption></figure><p>This is the “interface” of a pipeline, that again, has similarities to the “interface” of a spacecraft. The pipeline runs autonomously, once it is launched. We may be able to intervene through the input data, as a way to probe or test the behaviour of the pipeline, and we can observe its output as well as the metrics. We can also control the job, but this is mostly on/off, and in some very specific cases, other features such as “draining” or “updating”.</p><p>Other than that, once launched, the pipeline is on its own. We can hope for the best. But hope is not a strategy. We need some design principles to improve our chances of success.</p><h3>Principle 1: Use a High-Level Language</h3><p>The Apollo Guidance computer was programmed using its own assembly language. In assembly, you need to work with the instruction set that the computer provides you, and consider all the quirks of the machine you’re working on.</p><p>During the braking phase of the moon landing procedure, the Apollo Guidance Computer was required to fire the rocket engine at the right time to slow down the lunar module so that it would land on the designated spot. How do you implement the vector and matrix operations required for these orbital calculations in assembly?</p><p>You don’t, not even in the 1960s. The Apollo team built an interpreter that could be programmed by a high-level language that was capable of vector and matrix operations, and that would “compile” to assembly in real time.</p><p>Let’s apply this principle to the design of data processing pipelines.</p><p><a href="https://beam.apache.org/">Apache Beam</a> has several implementations in different languages, use the language that better fits your case, don’t worry about specific features of Beam being available in one or another language, but beware of I/O connectors, not all languages have the identical support. <strong>Cross language pipelines</strong> can help here. See for instance this workshop on <a href="https://spotify.github.io/scio/">Scio</a>, Spotify’s Scala SDK for Beam:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcGvaQp_h5ek%3Flist%3DPL4dEBWmGSIU9OkXQU2OAXmITPLhiMSPRp&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcGvaQp_h5ek&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcGvaQp_h5ek%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/90cbc0932439731b0d0000672c125960/href">https://medium.com/media/90cbc0932439731b0d0000672c125960/href</a></iframe><h3>Principle 2: Divide your program into jobs</h3><p>Dividing your program into jobs simplifies the program, enhances reusability of code, reliability and flexibility. And these were desirable properties in 1969, as they are today.</p><p>The Apollo Guidance Computer (AGC) was running an operating system that was “collaborative real-time”, which is quite different from modern operating systems that can run multiple programs at the same time. In the AGC, each job that was running was required to check every 20 ms if there was a new job ready to run. If there was, it had to relinquish control and allow the other job to run. This required all the jobs to contain code to check this register regularly. This is error-prone, but it was manageable, because the whole system was written by one team and they were not allowed to <a href="https://www.urbandictionary.com/define.php?term=WFH">WFH</a>.</p><p>When unexpected problems arise, their blast radius is limited if all the work is divided into jobs, compared to having one monolithic program for everything. If we look at the structure of each job, or pipeline, then Beam offers the <a href="https://beam.apache.org/releases/pydoc/2.2.0/apache_beam.transforms.ptransform.html">PTransform</a> to group differently operations that together implement a higher-level operation. For example, two I/O operations of different files and a join, could be a PTransform called “prepare input data”. As a design principle, we recommend to prefer simpler pipelines using higher-level PTransforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1bx0sE20C4G2-3wpPTjzw.png" /><figcaption>Use PTransforms to group individual transformation and make your pipeline easier to understand and maintain.</figcaption></figure><p>More in general, take a look at the job graph and see if there is a simpler way to achieve the objective. If you can simplify, this will make the job easier to understand and maintain. The Apache Beam documentation as has more <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/">guidance on this aspect of pipeline design</a>.</p><h3>Principle 3: Restart on failure</h3><p>Failures happened more than you might have thought during the famous Apollo 11 mission. One of the most critical phases of the mission is the final approach to the moon. This the phase where supposedly, Commander Armstrong seized manual control and saved the mission. Where does that story come from?</p><p>During the final phases, the computer is under a fairly heavy load of 80% due to the constant orbit recalculations as the lunar module descends towards the surface. Buzz Aldrin, the pilot, at that point decided to request the computer to also calculate the predicted landing site and continuously update this information and show it on the screen. This puts an extra 10% load on the computer, so we’re now at 90% load. Unfortunately, due to a hardware bug of one part of the spacecraft, there was an unexpected additional load of 15% and the computer got overloaded, triggering errors. The way for the computer to deal with this is to terminate all jobs and restart the essential ones. This worked.</p><p>But in the final descent phase, the auto pilot system actually generates 85% load. And with the 15% load from the hardware bug, the computer was overloaded even without Buzz Aldrin requesting updates of the landing site. But when Armstrong switched off the autopilot system to a more manual mode of operation, the problem went away. We say “more manual” because even in this mode, there is definitely no hardwired connection between Armstrong’s manual controller and the thrusters, there is always code in between the manual controller (the joystick) and the thrusters that move the spacecraft around.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QaZr67pxC9ASVzHQ.jpg" /><figcaption>Programming the Apollo Guidance Computer magnetic rope memory. Programming the computer was done by storing ones and zeroes in its memory, which was done by weaving a wire through a ring for a one, or not, for a zero. Source: <a href="https://airandspace.si.edu/multimedia-gallery/13006hjpg">National Air and Space Museum</a></figcaption></figure><p>In a data processing pipeline, we want complete jobs to restart on failure, as well as individual pipeline steps. Especially in the current situation where many workloads run on a public cloud, the environment is not predictable. You should count on workers disappearing, because the physical node fails, or because the datacenter operator decides it needs to be taken offline, or any other reason. You can’t let the whole workload fail every time that happens.</p><p>Pipelines need to be built with resilience to failures taken into consideration. This issue is acute in streaming pipelines because of the typically long running times. Specifically, we need to consider whether we will be draining our streaming pipeline, and process the data we have in flight, or if we will cancel the pipeline and kill the workers. In the case of canceling, it will be faster, but it may lead to some data loss. If we drain the pipeline, we will not lose data, but since we need to trigger processing of the open windows, we may have incomplete windows in our output and we will have to account for this later on. The Google Cloud Architecture Center also has some <a href="https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-deploying#updating_streaming_pipelines_in_production">guidance on this topic</a>.</p><p>At the Beam Summit 2022, <a href="https://twitter.com/MazlumTosun3">Mazlum Tosun</a> presented an interesting tool to simplify error reporting from your pipeline using the <a href="https://github.com/tosun-si/asgarde">Asgarde library</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0hPunokBPEE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0hPunokBPEE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0hPunokBPEE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/22d233d474565aead3fab9a2d7300552/href">https://medium.com/media/22d233d474565aead3fab9a2d7300552/href</a></iframe><h3>Principle 4: Checkpoint good state</h3><p>In the final phases of descent towards the moon surface, we explained that problems started occurring due to a hardware issue and jobs were being restarted. This started at about 10 miles above the surface. And it continued to happen as Aldrin and Armstrong approached the moon surface. The alarms were due to lack of capacity to run the jobs that the computer needs to run, like recalculation of the landing site location. The computer can safely deal with this by canceling the jobs and restarting them, but it needs to do it fast! This is where checkpointing existing good state comes in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rfru-pexMMc6e5Yx.jpg" /><figcaption>The Apollo Inertial Measurement Unit allows the computer to knows the orientation of the spacecraft, by reading out its sensors and performing a calculation. The result of this calculation can be checkpointed for faster recovery. Source: <a href="https://commons.wikimedia.org/w/index.php?curid=82248569">ArnoldReinhold — Own work</a>, CC BY-SA 4.0.</figcaption></figure><p>Similarly in a cloud environment, when your data processing jobs are restarted, you want the code to resume where it left off. We recommend to use checkpoints in your pipelines so that failed steps can recover from that point, and you don’t need to rerun the whole pipeline.</p><p>How to set or trigger a checkpoint is dependent on the implementation of the Beam Runner. Portable runners, such as <a href="https://cloud.google.com/dataflow">Dataflow</a> and <a href="https://flink.apache.org/">Flink</a>, will set a checkpoint in I/O operations. This is convenient, because when an error happens, you do not have to pull the data again from the source. They also set a checkpoint when a shuffle is initiated. Shuffling will happen prior to, for example, “group by key” and combiner steps, but sometimes you do not have an excuse to initiate a shuffle. In this case you can force a checkpoint by adding the <a href="https://beam.apache.org/releases/javadoc/2.5.0/org/apache/beam/sdk/transforms/Reshuffle.html">Reshuffle operation</a>. This operation is marked as deprecated in the Java API, but not becasue it will disappear. This is runner-dependent behaviour that has leaked into the Beam programming model.</p><p>Shuffle can also happen in I/O connectors, and a shuffle is guaranteed in a Splittable DoFn, that were treated extensively in a hands-on workshop at the Beam Summit 2022.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVQdtaaWxN0Y%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVQdtaaWxN0Y&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVQdtaaWxN0Y%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/61ab382a997c085c41596ce93427bcbd/href">https://medium.com/media/61ab382a997c085c41596ce93427bcbd/href</a></iframe><h3>Principle 5: Hardware monitors the software</h3><p>The Apollo Guidance computer is running its own custom OS that implements “collaborative multitasking”, as mentioned before. This requires that jobs are “collaborative”, meaning that they regularly check for the existence of other jobs (every 20 ms) and let them take control if there are any. It works well, but what if a job is stuck in an infinite loop and never checks again? The hardware is monitoring for this and after some longer time-out, the hardware will trigger a reboot of the system. A bit radical, but it often solves the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*SQTNn7PaVFLzRJzJ" /><figcaption>Margaret H. Hamilton with a stack of books containing Apollo Computer software, around 1969. She was awarded the Presidential Medal of Freedom in 2016 for her contribution. Source: <a href="https://mitmuseum.mit.edu/collections/object/GCP-00009689">MIT Museum</a></figcaption></figure><p>In our cloud-based data processing pipelines, we need a similar robustness. We can achieve this observability, to some extent, with the metrics provided by default by the runner. The Google Cloud Architecture has extensive guidance on <a href="https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-monitoring">how to do this using Dataflow</a>.</p><p>Dataflow also has great <a href="https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-planning#defining_and_measuring_slos">support for the montoring of “service level objectives”</a> (SLO’s), that you can use to agree SLA’s with the business owner of the process. The business owner is the ultimate decision maker in most cases, and they will have to balance a higher SLO against the higher cost of operation, and also other costs such as reduced frequency of updates. Google offers some very interesting <a href="https://sre.google/resources/practices-and-processes/art-of-slos/">free training material on SLO setting</a> to work on these questions with business and technical stakeholders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*cN2W0nV_sw7tF4ba" /><figcaption>An SLI, service level indicator, is a metric describing the performance of a pipeline. An SLO is a threshold for this SLI, and an SLA is contract that defines the consequences of breaching the SLO. Source: <a href="http://sre.google/">Google</a></figcaption></figure><h3>Principle 6: Send telemetry</h3><p>The user interface to the computer for the astronauts in the lunar landing module was the DSKY (pronounced “disky”). This would show a two-digit code for a verb and a two-digit code for a noun allowing the computer to convey what it was “thinking”, and the same for the astronauts to input commands to the computer. There were also about 4 additional numbers that the computer could use to provide information, like if you asked for the speed, or the altitude, or the coordinates of the predicted landing site, it could give you those numbers. That was a low-bandwidth communication channel, even for those days.</p><p>When the computer started showing error codes, there wasn’t much that the astronauts could find out about the cause. How could they know that it was safe to carry on and that they didn’t have to abort the mission?</p><p>The computer sends a much more complete dataset to Earth, to mission control in Houston, for the different experts there to analyse in real time. And this dataset, the telemetry, allowed them to verify that the computer was operating normally and that the astronauts could safely proceed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yNfDljh44uggxw-H.jpg" /><figcaption>The Apollo mission control room in 1969. Source: TRW/PHOTOQUEST/GETTY IMAGES</figcaption></figure><p>How do we send this kind of telemetry from a Beam pipeline? Beam implements three kinds of metrics: <a href="https://beam.apache.org/releases/pydoc/2.6.0/apache_beam.metrics.metric.html">Counter, Distribution and Gauge</a>. Not all runners implements all of these, so check the <a href="https://beam.apache.org/documentation/runners/capability-matrix/what-is-being-computed/">capability matrix</a> (under “Metrics”). We can use these metrics to communicate information about what’s happening in the pipeline. For example, the latency in pulling data from a source can be reported with it’s min and max, the mean and so on. We can also provide even more insight by defining “business level” metrics, rather than just hardware and systems metrics.</p><h3>Conclusions</h3><p>Launching a Beam pipeline into the cloud is not quite the same as launching a spaceship with people inside to the Moon, but we want both to be successful in the autonomous execution of their missions. The requirements to make them successful certainly have similarities.</p><p>In this post we have taken 6 design principles that were applied to the Apollo missions to the Moon, and we have applied them to the design of data processing pipelines using Apache Beam. There is, a host of other guidelines that one can apply to improve the quality of the data processing pipeline, for example:</p><ul><li><a href="https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-planning">Building production-ready data pipelines using Dataflow: Planning data pipelines</a> (Google Cloud Architecture Center)</li><li>The Apache Beam documentation has <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a>, <a href="https://beam.apache.org/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a> and <a href="https://beam.apache.org/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li><li>Google’s <a href="http://sre.google/">Site Reliability Engineering page</a> has a wealth of information, including free books, training material and use cases.</li></ul><p>Apply these principles bit by bit, and we assure you that the reliability and operating cost of your pipelines will go down over time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20f3d0e0760c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/houston-we-have-a-problem-six-apollo-mission-principles-for-pipeline-design-20f3d0e0760c">Houston, we have a problem: Six Apollo Mission Principles for Pipeline Design</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avoid Public PyPI Using Google Cloud Artifact Registry]]></title>
            <link>https://medium.com/google-cloud/avoid-public-pypi-using-google-cloud-artifact-registry-4464325a5e75?source=rss-94fa7ea880d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4464325a5e75</guid>
            <category><![CDATA[configuration-management]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[pypi]]></category>
            <category><![CDATA[artifact-registry]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Paul Balm]]></dc:creator>
            <pubDate>Tue, 23 Aug 2022 12:13:43 GMT</pubDate>
            <atom:updated>2022-08-25T16:37:24.928Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Why would you want to avoid <a href="https://pypi.org/">PyPI</a>? PyPI is great, it’s a massive bazaar with close to 400,000 packages, and publicly available since 2003. But unfortunately, it also has very limited control over what is published. It’s easy for people with bad intent to publish packages with very similar names to existing popular packages, to take advantage of common typo’s, and include malware. It’s also possible to find packages that the owner no longer has time to look after, and to attempt to hack the owner’s account and publish a new poisoned version of a valid existing package. There are many variations exploiting these <a href="https://cloud.google.com/resources/delivering-software-securely-whitepaper">software supply-chain vulnerabilities</a>, but we can also protect ourselves, by avoiding software repositories over which we have no control.</p><p>Larger organizations will often go further, prohibiting access to the public internet outright. Now if you need to install a Python package, what do you do? If you’re on Google Cloud, you use Artifact Registry.</p><figure><img alt="A cartoon of a cloud with a lock on it." src="https://cdn-images-1.medium.com/max/1024/0*ZRCPVTUSVJrmOU-O" /></figure><p>There is a lot of information about setting up Python repositories using Artifact Registry, typically to share internally developed Python packages. But the specific challenges for those in an environment without internet access are rarely covered. This post provides the detail on how to overcome those obstacles.</p><p>If you set up a private package repo, as we suggest in this post, you will then have to maintain it by continuously installing new versions of packages, to allow you users to stay up to date. This burden will be diminished when Artifact Registry makes its <a href="https://cloud.google.com/artifact-registry/docs/repositories/virtual-repo">virtual repositories</a> generally available, but until then, we will have to maintain by hand.</p><h3>Setting up a Python Index on Artifact Registry</h3><p>Your concerned administrator will lock down your internet access, for example by disallowing you from creating resources with an external IP address, such as virtual machines in Compute Engine, or Jupyter notebooks in Vertex AI Workbench. If the network (the VPC) of the resource does not have Cloud NAT enabled, it will not have internet access.</p><p>In this situation, when we need to install Python packages, we’re going to install them from a Python repository in Artifact Registry. We start by selecting Python packages and versions that we trust and that we will allow in our project. We will upload these packages to Artifact Registry, and the users of the project can install them when they need them.</p><p>But if we don’t have internet access, how can we get the packages to upload them to Artifact Registry? To set up our Python package repository, we will need internet access from somewhere. We could use a virtual machine in Compute Engine, that an administrator is allowed to create with an external IP address, or a VM that is created in a VPC that has Cloud NAT, to enable the internet access. In this case, we will have to block other users creating resources in this VPC, of course!</p><p>For the remainder of this post, we will assume that we are working on a virtual machine in the secured project, that has access to the public internet and to the Artifact Registry in our project. We will be using the <a href="https://cloud.google.com/sdk/docs/install">gcloud command-line interface</a>.</p><p>In this situation we can simply follow <a href="https://cloud.google.com/artifact-registry/docs/python">the documentation</a> to create a Python repository and upload our packages:</p><ol><li>Start by creating a Python repository called “medium” in the region of your choice:</li></ol><pre>gcloud auth login<br>gcloud config set project my_project<br>gcloud artifacts repositories create medium \<br>   --repository-format=python --location=europe-west1</pre><p>2. Often the list of required Python packages for a project is kept in a <a href="https://learnpython.com/blog/python-requirements-file/">requirements.txt file</a>. If you don’t have that, then you can create a requirements.txt using pip freeze:</p><pre>pip freeze &gt; requirements</pre><p>3. Ready to download all our packages! Or perhaps we should first check that our list doesn’t contain any known vulnerabilities… I recommend using <a href="https://www.redhat.com/sysadmin/find-python-vulnerabilities">pip-audit</a>. If it comes back with “no vulnerabilities found”, then we are good to go! You use pip-audit as follows:</p><pre>pip install virtualenv<br>virtualenv venv<br>source venv/bin/activate<br>pip install --upgrade pip-audit<br>pip-audit --requirement requirements.txt</pre><p>4. Ready to download all our packages, finally. The pip tool is helpful here:</p><pre>mkdir dist<br>pip download --destination-directory dist -r ../requirements.txt</pre><p>5. And now we upload the downloaded packages using twine, as outlined in the <a href="https://cloud.google.com/artifact-registry/docs/python/manage-packages">Google Cloud documentation</a> (don’t forget to replace “my_project” with the name of your project!)</p><pre>pip install twine<br>twine upload --repository-url <a href="https://europe-west1-python.pkg.dev/my_project/medium/">https://europe-west1-python.pkg.dev/my_project/medium/</a> dist/* --skip-existing</pre><p>If twine hangs on a large number of packages to upload, upload them one by one:</p><pre>for F in dist/*; do twine upload — repository-url <a href="https://europe-west1-python.pkg.dev/my_project/medium/">https://europe-west1-python.pkg.dev/my_project/medium/</a> $F — skip-existing; done</pre><p>Congratulations, your Python repository is live!</p><h3>Using your private Python package repository</h3><p>The <a href="https://cloud.google.com/artifact-registry/docs/python/authentication">documentation</a>, and many blogs and stackoverflow questions, will tell you to do two things: To set up authentication and to configure pip to retrieve packages from your Python repository in Artifact Registry. Let’s look at these two steps.</p><figure><img alt="A cartoon of a traffic light on yellow." src="https://cdn-images-1.medium.com/max/1024/1*oHxA_t5Ih583w_u6UZ4e7Q.png" /></figure><h3>Setting up authentication</h3><p>Before installing packages from your repository in Artifact Registry, using pip install, you need to enable pip to authenticate with Artifact Registry. Pip is not a Google Cloud aware tool, so it needs some mechanism to retrieve the Google Cloud credentials. This mechanism is the “keyring” package with Google Cloud Artifact Registry plug-in. You install the keyring and the plug-in using pip install.</p><p>But wait… If I don’t have internet access and I can’t authenticate with my private repository yet, how do I install this keyring with the plug-in? The snake bites its own tail.</p><h4>The easiest way</h4><p>The easiest solution to this problem is to have an administrator build a custom container. If this container build process has access to the internet, then you can just pip install the packages as part of the Docker build:</p><pre>RUN pip install keyring==23.7.0<br>RUN pip install keyrings.google-artifactregistry-auth==1.0.0</pre><p>Note that this step does involve installing these packages from the public PyPI, which is what we’re trying to avoid. But we need to download these packages at some point, and as long as it’s controlled by an administrator, we can, for instance, unpublish the container if any contamination of these packages would have occurred.</p><h4>If the easiest way doesn’t apply</h4><p>The easiest way involves building a custom image and using internet access during the container build process. If you build your container using Cloud Build, then the Cloud Build workers must have internet access. If any of these two conditions are violated, we need to look for another solution.</p><p>The solution is for the administrator to download the packages, as we have done previously with the packages from our requirements.txt file. We install the downloaded files using pip, which will enable the authentication with Artifact Registry.</p><p>We just need to install two packages: keyring and keyrings.google-artifactregistry-auth. But we are also going to need their dependencies. And the dependencies of the dependencies. Not to mention, <em>their</em> dependencies. You get the point. Pip download will take care of downloading everything. These are 22 packages in total, that you need to install in the right order. For current versions, keyring 23.7.0 and keyrings.google_artifactregistry_auth 1.0.0, this is the list, in a valid order:</p><pre>jeepney-0.8.0-py3-none-any.whl<br>typing_extensions-4.3.0-py3-none-any.whl<br>zipp-3.8.1-py3-none-any.whl<br>importlib_metadata-4.12.0-py3-none-any.whl<br>pycparser-2.21-py2.py3-none-any.whl<br>cffi-1.15.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl<br>cryptography-37.0.4-cp36-abi3-manylinux_2_24_x86_64.whl<br>SecretStorage-3.3.2-py3-none-any.whl<br>keyring-23.7.0-py3-none-any.whl<br>pyasn1–0.4.8-py2.py3-none-any.whl<br>pyasn1_modules-0.2.8-py2.py3-none-any.whl<br>rsa-4.9-py3-none-any.whl<br>cachetools-5.2.0-py3-none-any.whl<br>six-1.16.0-py2.py3-none-any.whl<br>google_auth-2.9.1-py2.py3-none-any.whl<br>certifi-2022.6.15-py3-none-any.whl<br>charset_normalizer-2.1.0-py3-none-any.whl<br>idna-3.3-py3-none-any.whl<br>urllib3–1.26.11-py2.py3-none-any.whl<br>requests-2.28.1-py3-none-any.whl<br>pluggy-1.0.0-py2.py3-none-any.whl<br>keyrings.google_artifactregistry_auth-1.0.0-py3-none-any.whl</pre><p>The administrator can share these packages via Cloud Storage, and whenever needed, the user can retrieve and install them. But this is a bit of a hassle, and if we are running any kind of distributed processing (using Dataflow, for example), we would need to go through this process every time a new worker starts up, which will affect processing speed. It would be better to generate a docker image, and use this image to start up new workers. I won’t describe that in detail here, but if you follow for example <a href="https://cloud.google.com/dataflow/docs/guides/using-custom-containers">Using custom containers in Dataflow</a>, you would download all the packages to a directory (called pydist for instance) and include this in your Dockerfile:</p><pre>ADD pydist pydist<br>RUN pip install pydist/jeepney-0.8.0-py3-none-any.whl<br>RUN pip install pydist/typing_extensions-4.3.0-py3-none-any.whl<br>RUN pip install pydist/zipp-3.8.1-py3-none-any.whl<br>...</pre><p>And so on.</p><p>Once we have these packages installed, we can run keyring — list-backends and it should output something like this:</p><pre>keyrings.gauth.GooglePythonAuth (priority: 9)<br>keyring.backends.chainer.ChainerBackend (priority: -1)<br>keyring.backends.fail.Keyring (priority: 0)</pre><p>And this means we are all set up. If you’re not authenticated already, use gcloud auth login to get your credentials. Now, if you run pip install, it will be able to authenticate with your repository Artifact Registry. That is, if pip has the configuration to know where your repository is. We’ll see how that’s done in a moment, but let’s reflect for a moment.</p><h3>A custom container with Everything We Need</h3><p>If we’re going to have to download 22 packages and possibly build a custom container, so that we can install packages from Artifact Registry… Why don’t we just create a custom container that has all the packages we need, so that we don’t need to install anything?</p><p>Building a custom container with all the code required is recommended practice from a performance point of view, and for reproducibility. Performance, because there is no installation work required when a container starts up. And reproducibility because the software versions on the container are fixed, whereas a requirements.txt file often leaves (some) package versions unspecified.</p><p>But to be able to do this, we’re going to need to know the complete list of packages that we’re going to need. When we find out we need to upgrade a package, we need to rebuild the container. Same when we need to add a package that we weren’t using before. If you have a Dataflow pipeline, or similar, that is working in production, then we would still recommend creating a custom container for this process. But if your team is developing new code, the management overhead of having to build a new container is going to be very high. For example, a notebook instance would have to be recreated on top of a newly built container, every time you want to install a Python package. So Artifact Registry makes the management of your Python packages a lot easier.</p><p>When you need to use a package that no one in your project has used before, your administrator just downloads it from PyPI, verifies that it has no vulnerabilities (!), and uploads it to Artifact Registry.</p><h3>Configure pip!</h3><p>You can configure pip by passing the right command-line arguments on every call, but this is rather error-prone. The <a href="https://console.cloud.google.com/artifacts">Artifact Registry UI</a> has a button for “Setup Instructions”</p><figure><img alt="Google Cloud Console showing the UI for Artifact Registry, which has a button labeled “Setup Instructions”." src="https://cdn-images-1.medium.com/max/848/0*gcU4dblT0z_yv1-O" /></figure><p>If you click it, it will tell you to run this command (again, replace “my_project” with your project name)</p><pre>gcloud artifacts print-settings python \<br>  --project=my_project --repository=medium --location=europe-west1</pre><p>The output of this command is some configuration that you can paste into your pip configuration file. It will tell pip to add an “extra index URL”. This will cause pip to look at the public PyPI and also at your local repository. This is useful, so you can publish your own packages internally and you don’t lose access to PyPI for everything else. Unfortunately, it doesn’t work for us.</p><p>If you don’t have access to the internet, pip will hang on trying to check PyPI. We don’t need to add an extra index URL, we need to set <em>the</em> index URL and tell pip not to look anywhere else. This would be the configuration for “my_project”, which also disable pip’s update check:</p><pre>[global]<br>index-url = <a href="https://europe-west1-python.pkg.dev/my_project/medium/simple/">https://europe-west1-python.pkg.dev/my_project/medium/simple/</a></pre><pre>[global]<br>disable-pip-version-check = True</pre><p>Store this information in a file called pip.conf and tell pip to use it:</p><pre>export PIP_CONFIG_FILE=pip.conf</pre><p>Use an absolute path if you want this to work from any directory. If you were building a custom container, then you’re also going to want to include this configuration in your Dockerfile:</p><pre>COPY pip.conf pip.conf<br>ENV PIP_CONFIG_FILE=pip.conf</pre><p>If you’re not building a custom container that has this configuration set up, then these steps have to be done every time after the container starts up. If you’re using a Vertex AI User Managed Notebook, then you can automate this using a “post-start-up script” for your notebook instance. You need to store the script in Cloud Storage and pass the Cloud Storage URL to the <a href="https://cloud.google.com/sdk/gcloud/reference/notebooks/instances/create#--post-startup-script">–post-startup-script argument</a>, for instance:</p><pre>gcloud notebooks instances create example-instance \<br>  --vm-image-project=deeplearning-platform-release \<br>  --vm-image-family=caffe1-latest-cpu-experimental \<br>  --machine-type=n1-standard-4 \<br>  --location=us-central1-b \<br>  --post-startup-script=gs://…</pre><p>You can now install packages using pip install as you would normally:</p><pre>pip install packagename</pre><p>Look for the first line of the output to verify that pip is looking at the correct repository:</p><pre>Looking in indexes: <a href="https://europe-west1-python.pkg.dev/my_project/medium/simple/">https://europe-west1-python.pkg.dev/my_project/medium/simple/</a></pre><h3>Conclusion</h3><p>If you have followed this far, then you are done. You now have configured pip to query your private Python repository in Artifact Registry. You have also configured it to be able to authenticate with the Artifact Registry, without ever being able to go out to the public internet!</p><p>The awareness of the security implications of relying on public repositories for software is growing. We hope this guide will help you configure a secure environment on Google Cloud.</p><figure><img alt="Cartoon: The sun comes up over a green field." src="https://cdn-images-1.medium.com/max/1024/0*3t3C239SsVXPyEX5" /></figure><h3>References</h3><ol><li><a href="https://cloud.google.com/resources/delivering-software-securely-whitepaper">Delivering software securely</a>, Google Cloud whitepaper</li><li><a href="https://cloud.google.com/artifact-registry/docs/how-to">Artifact Registy documentation</a></li><li><a href="https://cloud.google.com/artifact-registry/docs/repositories/virtual-repo">Creating virtual repositories on Artifact Registry</a>, Artifact Registry documentation. Feature in private preview as of August 2022.</li><li><a href="https://cloud.google.com/artifact-registry/docs/python/manage-packages">Managing Python packages</a>, Artifact Registry documentation</li><li><a href="https://www.redhat.com/sysadmin/find-python-vulnerabilities">How to find third-party vulnerabilities in your Python code</a>, Red Hat. Instructions for pip-audit.</li></ol><p>—</p><p>Image credit: Denise Balm</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4464325a5e75" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/avoid-public-pypi-using-google-cloud-artifact-registry-4464325a5e75">Avoid Public PyPI Using Google Cloud Artifact Registry</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>