<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thakurswati on Medium]]></title>
        <description><![CDATA[Stories by Thakurswati on Medium]]></description>
        <link>https://medium.com/@thakurswati9?source=rss-4f88bd511453------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Zsu3ccjROl1dIWBu-ImLJg.jpeg</url>
            <title>Stories by Thakurswati on Medium</title>
            <link>https://medium.com/@thakurswati9?source=rss-4f88bd511453------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:51:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thakurswati9/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlocking Insights with Multimodal Vector Search in BigQuery — Part 2]]></title>
            <link>https://medium.com/google-cloud/unlocking-insights-with-multimodal-vector-search-in-bigquery-part-2-68a05653551b?source=rss-4f88bd511453------2</link>
            <guid isPermaLink="false">https://medium.com/p/68a05653551b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[multimodal-learning]]></category>
            <dc:creator><![CDATA[Thakurswati]]></dc:creator>
            <pubDate>Tue, 03 Dec 2024 12:42:10 GMT</pubDate>
            <atom:updated>2024-12-04T09:42:32.784Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unlocking Unstructured Data Insights: Multimodal Search in BigQuery- Part 2</h3><p>Unlock the hidden potential of your unstructured data with BigQuery’s multimodal vector search.Whether you’re analyzing product images, identifying trends, or building AI-powered applications, BigQuery’s multimodal capabilities can help you unlock new levels of insight and innovation.</p><p>In <a href="https://medium.com/google-cloud/unlocking-insights-with-multimodal-vector-search-in-bigquery-part-1-a560b2183b83">Part 1</a> of the blog series, we explored the exciting possibilities of multimodal vector search. Now, let’s put theory into practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kWGMveqOIxIxVZw7OKON1A.jpeg" /></figure><p>This step-by-step tutorial will show you how to leverage this cutting-edge technology to perform advanced analysis and gain a deeper understanding of your visual content. Get ready to discover new insights and build innovative applications with the power of BigQuery.</p><p><em>Imagine you’re an e-commerce company with thousands of product images stored in Google Cloud Storage (GCS). You want to quickly identify if a new product being considered has any visual similarities to products you’ve sold in the past. This can help you understand market trends, predict potential sales, and avoid redundant inventory.</em></p><h3>How it Works in BigQuery</h3><p>BigQuery utilizes powerful machine learning models to generate embeddings for your data. These embeddings capture the semantic essence of your content, whether it’s text, images, or other modalities. These embeddings are then stored as vectors within BigQuery.</p><p>When you perform a search, BigQuery uses a vector search algorithm to find the data points whose vectors are closest to the vector of your query. This ensures that the results are relevant to your search intent, even if they don’t contain the exact keywords.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L-w_pecRYKQ77qon" /></figure><h3>Let’s get started!</h3><h3>Prerequisite</h3><ol><li>Enable the <strong>BigQuery API</strong>.</li><li>Upload a Sample Images in your GCS bucket</li><li>Here used : gs://&lt;project_name&gt;_products_1667313204/sample/ sample.pdf</li></ol><p>Create new <strong>External source Connection</strong> in Bigquery</p><ol><li>Click Add</li><li>Connections to external Source</li><li>Connection Type as Vector remote models,remote functions…</li><li>Connection id : <em>vector_conn</em></li><li>Click <strong>Create Connection</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ex4VlWYImiqkEx51ZqPcw.png" /><figcaption>Crearte a new External source Connection</figcaption></figure><ol><li>Navigate to new connection &amp; Copy the new service account id</li><li>&lt;service-account-name&gt;@gcp-sa-bigquery-condel.iam.gserviceaccount.com</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f8G03WbOtckJs6RaM1W7kQ.png" /></figure><p>Grant IAM permissions to the copied service account</p><ol><li>Storage object viewer</li><li>Vertex ai viewer</li></ol><h3>Vector Search</h3><ol><li><strong>Create a new BQML model:</strong> Since our data resides in a GCS bucket in an unstructured format, we’ll designate a multimodal model as the endpoint to handle this type of data effectively.</li></ol><pre>/* Step 1 Create a BQML model for Embeddings */<br>create or replace model<br>`my-test-world-ts.bqml_tutorial.mm_embedding`<br>remote with connection `my-test-world-ts.us.vector_conn`<br>options(<br>endpoint = &#39;multimodalembedding@001&#39;<br>);</pre><p><strong>2. Create an object table: </strong>Object tables let you analyze unstructured data in Cloud Storage.Read more about object tables <a href="https://cloud.google.com/bigquery/docs/object-table-introduction">here</a></p><p><strong>Note:</strong> <em>Object tables only store the metadata &amp; there is no movement of data from GCS to BQ</em></p><pre>/* Step 2 Create an Object Table */<br>create or replace external table `my-test-world-ts.bqml_tutorial.image_object_table`<br>with connection `my-test-world-ts.us.vector_conn`<br>options<br>(<br>object_metadata=&#39;SIMPLE&#39;,<br>uris=[&#39;gs://my-test-world-ts_products_1667313204/sample/images/*&#39;]<br>)</pre><p><strong>3. Create Embeddings Table</strong> :</p><p>Use <em>ml.generate</em> embeddings function to create embeddings.</p><ol><li>Input for the function-</li></ol><ul><li>Remote embedding <em>model name</em>(created in step 1)</li><li><em>Table name</em> to reference objects in GCS(created in step 2)</li></ul><p>Behind the scenes Bigquery is submitting the document data in the object table to multimodal embeddings API hosted in Vertex AI.And the returned results are then stored in Bigquery table</p><p>More about ml.generate embedding function <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-embedding#syntax">here</a></p><pre>/* Step 3 Create a Table with Embeddings */<br>create or replace table `my-test-world-ts.bqml_tutorial.retail_store_embeddings`<br>as select *,REGEXP_EXTRACT(uri, r&#39;[^/]+$&#39;) as product_name<br><br><br>from ML.GENERATE_EMBEDDING<br>(<br>MODEL `my-test-world-ts.bqml_tutorial.mm_embedding`,<br>TABLE `my-test-world-ts.bqml_tutorial.image_object_table`<br>)<br></pre><p>4. Verify the new embeddings table</p><pre>/* Step 4 Validate the Table with Embeddings */<br>Select * from `my-test-world-ts.bqml_tutorial.retail_store_embeddings`</pre><p>5. Run the <strong>Vector Search </strong>on the product images catalog in GCS to find if such products have been sold in the past</p><ul><li>Uses <em>ml.generate_embedding</em> function to generate embedding using the remote multimodal model</li><li>It then takes the embedding &amp; compares it to all the embeddings created in step 3 table</li><li>Vector search systems typically use distance metrics, such as cosine distance to compare vectors and rank them by similarity.In this case, we’ll use this technique to retrieve the top 2 most similar product images from GCS.</li></ul><pre>/* Step 5 Run a Vector Search */<br>select base.uri,<br>base.product_name,<br>base.content_type,<br>distance<br> from<br>vector_search(table &lt;dataset_name&gt;.retail_store_embeddings,&#39;ml_generate_embedding_result&#39;,<br>(<br>SELECT ml_generate_embedding_result as embedding_col<br>FROM<br> ML.GENERATE_EMBEDDING<br> (<br>   MODEL `&lt;dataset_name&gt;.mm_embedding`,<br>   (select &#39;Men Sweaters&#39; as content),<br>   STRUCT(TRUE AS flatten_json_output)<br> )<br>),top_k =&gt; 2,<br>distance_type =&gt; &#39;COSINE&#39;<br>);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0XHYzRoaUIZI3ts12XHVOg.png" /></figure><p>6. [Optional ] Create Vector Index on embeddings table for faster &amp; scalable semantic search</p><pre>/* Create Index on Embeddings table */<br>CREATE VECTOR INDEX image_index ON &lt;dataset_name&gt;.retail_store_embeddings(ml_generate_embedding_result)<br>OPTIONS(distance_type=&#39;COSINE&#39;, index_type=&#39;IVF&#39;, ivf_options=&#39;{&quot;num_lists&quot;: 10}&#39;);</pre><p>With Embeddings generation &amp; Vector Search built in Bigquery this process is highly simplified.</p><p>Multimodal vector search in BigQuery empowers you to unlock the full potential of your data. By combining the power of vector search with BigQuery’s scalability and performance, you can build innovative applications that deliver exceptional user experiences.</p><p>Hope this blog helps you in applying these techniques to your own datasets and discover the hidden connections within your data.</p><p>Start exploring the possibilities today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68a05653551b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/unlocking-insights-with-multimodal-vector-search-in-bigquery-part-2-68a05653551b">Unlocking Insights with Multimodal Vector Search in BigQuery — Part 2</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Insights with Multimodal Vector Search in BigQuery — Part 1]]></title>
            <link>https://medium.com/google-cloud/unlocking-insights-with-multimodal-vector-search-in-bigquery-part-1-a560b2183b83?source=rss-4f88bd511453------2</link>
            <guid isPermaLink="false">https://medium.com/p/a560b2183b83</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[vector-search]]></category>
            <dc:creator><![CDATA[Thakurswati]]></dc:creator>
            <pubDate>Mon, 02 Dec 2024 02:47:07 GMT</pubDate>
            <atom:updated>2024-12-04T06:04:26.786Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unlocking Unstructured Data Insights: Multimodal Search in BigQuery- Part 1</h3><p>Unlock the hidden potential of your unstructured data with BigQuery’s multimodal vector search.</p><p>Imagine you’re an e-commerce organization, and you want to analyze your product catalog images to make better decisions for this year’s catalog. You might be wondering, <strong><em>“Is it even possible to analyze unstructured data like images in BigQuery?”</em></strong></p><p>The answer is a resounding yes!</p><p>BigQuery, Google Cloud’s serverless, highly scalable, and cost-effective multicloud data warehouse, now has a game-changing feature: <em>M</em><strong><em>ultimodal Vector Search</em></strong>. This powerful capability allows users to search and analyze data using not just text, but also images and embeddings generated from other modalities like audio and video.</p><p><em>In this two-part blog series, we’ll delve into the exciting possibilities unlocked by multimodal vector search in BigQuery, including a step-by-step tutorial</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y81dXkY7rtb_pSIWKuqXdg.png" /></figure><p><strong>The Growing Need for Unstructured Data Analysis</strong></p><p>With the ever-increasing volume of data in the digital world, unstructured data holds a wealth of insights. There’s a huge need to perform analysis using this unstructured data, including images.</p><p>But is it possible to analyze images simply by providing a text query? Yes, it is! <strong>Now you can analyze unstructured data using vector search in BigQuery.</strong></p><p><strong>Traditional Search vs. Multimodal Vector Search</strong></p><p>Before diving into that, let’s first understand the difference between <strong><em>traditional search</em></strong> and <strong><em>multimodal vector search</em></strong>.</p><ul><li><strong>Traditional search</strong> performs exact keyword matching from user queries to the data to retrieve relevant results.</li><li><strong>Vector search</strong> takes a different approach by seeking out related objects that share similar characteristics or semantics. You can think of it as finding results based on meaning or understanding rather than just exact wording.</li></ul><p>To perform a vector search, the content needs to be represented as vectors/embeddings by AI models-These embeddings represent the data in a multidimensional vector space.</p><p>Once these vector representations are generated, they are essentially sets of numbers, making them easy to compute with. For instance, instead of searching for a specific word in text, the goal is to find the closest vector (from the text embeddings) to the query vector (representing the word we’re searching for). This process relies on well-established vector computing methods, such as calculating the distance between vectors or minimizing the angle between them (cosine similarity).</p><p><strong>What is Multimodal Vector Search?</strong></p><blockquote>Multimodal vector search takes this concept further by combining different data types (text, images, audio, etc.) into a single vector space. This allows you to search across different modalities and find items that are similar in meaning, regardless of their format.</blockquote><p>Multimodal vector search makes this possible by embedding both text and images in the same vector space, allowing you to search across both modalities. Multimodal embedding is the process of generating a vector representation of an image (or any structured or unstructured data) that captures its features and characteristics.</p><p><strong>Lets now understand this concept with a real world example</strong></p><p><strong>Scenario: Searching for “cozy winter cabin” on a hotel website.</strong></p><p><strong>Keyword Search:</strong></p><p><strong>How it works:</strong> The search engine looks for listings that contain the exact words “cozy,” “winter,” and “cabin.”</p><p><strong>Results:</strong> You might get a list of cabins that explicitly mention those words in their descriptions. However, you might miss out on listings that describe themselves as “rustic,” “charming,” or “warm” even though they offer a similar experience.</p><p><strong>Limitations:</strong></p><ul><li><strong>Synonyms and related terms:</strong> It doesn’t understand that “rustic” or “charming” could also mean “cozy.”</li><li><strong>Context and meaning:</strong> It doesn’t grasp the overall concept of a “<strong>cozy winter cabin</strong>” and might show pullovers that are cozy but not suitable for winter.</li></ul><p><strong>Vector Search:</strong></p><p><strong>How it works:</strong> The search engine analyzes the meaning and context of your query and compares it to the semantic meaning of the listings’ descriptions and images.</p><p><strong>Results:</strong> You’ll likely get a wider range of results that capture the essence of a “cozy winter cabin,” including those that:</p><ul><li>Use synonyms like “rustic” or “charming.”</li><li>Feature images of fireplaces, snowy landscapes, and warm interiors.</li><li>Mention amenities like hot tubs or wood-burning stoves.</li></ul><p><strong>Advantages:</strong></p><ul><li><strong>Captures meaning:</strong> Understands the intent behind your search, not just the keywords.</li><li><strong>Handles synonyms and related terms:</strong> Finds relevant results even if they don’t use the exact same words.</li></ul><p><strong>Considers multiple modalities:</strong> Can analyze images and text together for a more comprehensive understanding.</p><p>This highlights how vector search can elevate the user experience, making it more intuitive, personalized, and ultimately more successful in helping you find what you need.</p><p>Image retrieval systems have traditionally used features extracted from the images, such as content labels, tags, and image descriptors, to compare images and rank them by similarity. However, vector similarity search offers several benefits over traditional keyword-based search and is becoming a vital component in popular content search services.</p><p>Now that we’re clear about these different types of search mechanisms, in the next part, we’ll learn how to perform multimodal vector search in BigQuery.</p><p><a href="https://medium.com/@thakurswati9/unlocking-insights-with-multimodal-vector-search-in-bigquery-part-2-68a05653551b"><strong>Checkout Part 2 for a step-by-step tutorial to implement multimodal vector search in Bigquery<em>!!!</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a560b2183b83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/unlocking-insights-with-multimodal-vector-search-in-bigquery-part-1-a560b2183b83">Unlocking Insights with Multimodal Vector Search in BigQuery — Part 1</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DVT — Google Cloud’s Data Validation Tool]]></title>
            <link>https://medium.com/google-cloud/dvt-google-clouds-data-validation-tool-bac307779635?source=rss-4f88bd511453------2</link>
            <guid isPermaLink="false">https://medium.com/p/bac307779635</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-validation]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[dvt]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Thakurswati]]></dc:creator>
            <pubDate>Mon, 15 May 2023 09:11:14 GMT</pubDate>
            <atom:updated>2023-05-16T02:12:48.063Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data validation made easy with — Google Cloud’s Data Validation Tool — DVT</h3><blockquote>Data migration is like <em>moving to a new house </em>— it requires careful planning, organization, and attention to detail. Just as you would double-check your belongings before moving, <strong>data validation</strong> plays a vital role in the <strong>migration process</strong>, providing assurance that your valuable data makes the move securely and error-free</blockquote><p>In this article, we’ll explore the different methods to execute data validation using Google Cloud’s data validation tool — DVT(<strong>locally &amp; through GCS</strong> ) and how it can help make your migration journey smoother.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cdi509ac6Bf_W7yOInQhyg.jpeg" /></figure><p>Well Data validation is not just a one-time activity in migration process; it’s an ongoing journey towards data excellence.But it can be a daunting task that takes up valuable time and resources.</p><p>That’s where Google Cloud’s <strong>data validation tool (DVT)</strong> comes in. Whether you’re migrating databases, applications, or other workloads to the cloud, DVT can help streamline your data validation process.</p><h3>What is DVT?</h3><p>DVT, an open sourced Python CLI tool that provides an automated and repeatable solution for validation across different environments. The tool uses the Ibis framework to connect to a large number of data sources including BigQuery, Cloud Spanner, Cloud SQL, Teradata, and more.</p><p>DVT supports following types validations:</p><ul><li>Column validation (count, sum, avg, min, max, group by)</li><li>Row validation (BQ, Hive, Teradata, Oracle, SQL Server, Postgres, Mysql only)</li><li>Schema validation</li><li>Custom Query validation</li><li>Ad hoc SQL exploration</li></ul><h3>Key Components</h3><ol><li><strong>Connections </strong>— Before running validations, DVT requires setting up a source and target connection. These connections can be stored as <strong>JSON files</strong> locally or in a GCS directory.</li><li><strong>Validations Config</strong> — DVT allows you to save or edit validation configurations in a <strong>YAML file</strong>. This is useful for running common validations or updating the configuration.YAMLs are not mandatory to execute the validation &amp; you can provide all the configs within CLI command as well.</li></ol><p>Lets get on the console now &amp; get our hands dirty!</p><h4><strong>a. Pre-requisites</strong> for using DVT :</h4><ol><li><strong>Install DVT:</strong> Follow the steps mentioned <a href="https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/installation.md">here</a></li><li>Install <strong>External Source driver package</strong>( required if using external source like SQL server, Teradata, DB2 etc.)</li></ol><h3>Method 1 — Configs in Local</h3><p>In Method 1, we’ll store connection &amp; validation details locally &amp; then execute the validation</p><h4>b. Create Source &amp; Target connections</h4><p><strong>Command Template:</strong></p><pre>data-validation connections add --secret-manager-type &lt;None|GCP&gt; --secret-manager-project-id &lt;SECRET_PROJECT_ID&gt; --connection-name CONN_NAME source-type</pre><p>Secret manager flags are optional</p><p>— secret-manager-type &lt;None|GCP&gt;<br> — secret-manager-project-id &lt;SECRET_PROJECT_ID&gt;</p><p><strong>Create Source(here SQL Server) connection:</strong></p><pre>data-validation connections add --connection-name MY_SQL_CONN MSSQL --host=&lt;host_name&gt; --port=&lt;port&gt; --user=&lt;username&gt; --password=&lt;password&gt; --database &lt;database&gt;</pre><p><strong>Create Target(here Bigquery) connection:</strong></p><pre>data-validation connections add --connection-name MY_BQ_CONN BigQuery --project-id MY_GCP_PROJECT</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tnjuqAA5aYYI1-wO1fg7UA.png" /><figcaption>Connection JSON files stored in default Local Directory path</figcaption></figure><h4>c. Create Validation config YAML</h4><p>Generate the YAML configuration file by specifying the — config-file flag and the YAML file you want to create.</p><pre>data-validation validate column -sc MY_SQL_CONN -tc MY_BQ_CONN -tbls &lt;src_tbl&gt;=&lt;tgt_tbl&gt; -c &lt;filename&gt;.yaml </pre><p>Here DVT will perform column validation with default aggregation type as ‘COUNT *’.</p><h4>d. Run the Validation</h4><p>Run the validation using above created YAML config file</p><pre>data-validation configs run -c &lt;filename&gt;.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9F7o0VVpDiwz38_LvlzVg.png" /><figcaption>DVT Validation execution using YAML file</figcaption></figure><p>You can execute validation without YAML files as well by providing all the configurations in CLI command</p><pre>data-validation validate column -sc my_bq_conn -tc my_bq_conn -tbls bigquery-public-data.new_york_citibike.citibike_trips</pre><h3>Method 2 — Configs in GCS</h3><p>In Method 2, we’ll not store connections &amp; validation details locally and access connections &amp; validation files directly from GCS bucket</p><h4>a. Set Environment Variable</h4><p>The validation config file is saved to the GCS path specified by the PSO_DV_CONFIG_HOME env variable if that has been set; otherwise, it is saved to wherever the tool is run.</p><pre>export PSO_DV_CONFIG_HOME=gs://my-bucket/my/connections/path/</pre><p><strong>Note that if this environment variable is set, query validation YAML configs &amp; connection JSONs both need to be present in GCS path(tightly coupled)</strong></p><h4>b. Create Source &amp; Target connections</h4><p><strong>Command Template:</strong></p><pre>data-validation connections add --secret-manager-type &lt;None|GCP&gt; --secret-manager-project-id &lt;SECRET_PROJECT_ID&gt; --connection-name CONN_NAME source-type</pre><p>Secret manager flags are optional</p><p>— secret-manager-type &lt;None|GCP&gt;<br> — secret-manager-project-id &lt;SECRET_PROJECT_ID&gt;</p><p><strong>Create Source(here SQL Server) connection:</strong></p><pre>data-validation connections add --connection-name MY_SQL_CONN MSSQL --host=&lt;host_name&gt; --port=&lt;port&gt; --user=&lt;username&gt; --password=&lt;password&gt; --database &lt;database&gt;</pre><p><strong>Create Target(here Bigquery) connection:</strong></p><pre>data-validation connections add --connection-name MY_BQ_CONN BigQuery --project-id MY_GCP_PROJECT</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PV55hFbcrAJPpxoxCizfGQ.png" /><figcaption>Connection JSON files saved in specified GCS path inside connections folder</figcaption></figure><p><strong>Note: Connection JSON files are created inside subfolder named ‘connections’ in </strong>PSO_DV_CONFIG_HOME <strong>variable specified GCS path</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NdZT4CrzxUNpbLsfqTp69w.png" /></figure><h4>c. Create Validation config YAML</h4><p>Generate the YAML configuration file by specifying the — config-file flag and the YAML file you want to create.</p><pre>data-validation validate column -sc MY_SQL_CONN -tc MY_BQ_CONN -tbls &lt;src_tbl&gt;=&lt;tgt_tbl&gt; -c &lt;filename&gt;.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P7zaFKNe4xpS613uifWBJw.png" /><figcaption>Validation YAML config file saved in specified GCS path inside validations folder</figcaption></figure><p><strong>Note: Config YAML files are created inside subfolder named ‘validations’ in </strong>PSO_DV_CONFIG_HOME <strong>variable specified GCS path</strong></p><h4>d.Run the validation</h4><pre>data-validation configs run -c &lt;filename&gt;.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TMCafOlRXk1qMCgv4M4-uA.png" /><figcaption>DVT execution using YAML config file in GCS path</figcaption></figure><p>And we are done.Hope this blog helps you to get started with a smooth validation process using DVT, see you next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bac307779635" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/dvt-google-clouds-data-validation-tool-bac307779635">DVT — Google Cloud’s Data Validation Tool</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Practices for Data Sharing On Analytics Hub]]></title>
            <link>https://medium.com/google-cloud/best-practices-of-data-sharing-on-analytics-hub-378d42719045?source=rss-4f88bd511453------2</link>
            <guid isPermaLink="false">https://medium.com/p/378d42719045</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analytics-hub]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[best-practices]]></category>
            <dc:creator><![CDATA[Thakurswati]]></dc:creator>
            <pubDate>Mon, 27 Feb 2023 11:16:16 GMT</pubDate>
            <atom:updated>2023-03-13T11:13:41.526Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Analytics Hub</strong> is a <strong>data exchange platform</strong> that enables you to share data and insights at scale across organizational boundaries with a robust security and privacy framework. With Analytics Hub, you can discover and access a data library curated by various data providers.</blockquote><p>If you’re familiar with Google Cloud Platform’s data sharing platform, Analytics Hub, you would likely know how effortlessly it allows you to share analytics-ready datasets without having to copy the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QUjf1E-ANwVRwg2Ctbcnwg.png" /><figcaption>Analytics Hub Publisher-Subscriber workflow</figcaption></figure><blockquote>As a data product owner, we want to ensure that only users who should have access to a specific dataset have permission to use that data.</blockquote><p>When sharing data, it is important to remember the key principle of enabling fine-grained access control. But at the same time subscriber must be able to access the data shared with them, without requiring an additional broader role.</p><p>In this blog, we will be discussing few of the important considerations that can help with architecting data sharing on Analytics Hub in a better way.</p><h4><strong>Share Views in Authorised Datasets</strong></h4><p>Sharing Views directly in a shared dataset from a publisher project might not let your subscriber access view’s data since the Subscriber will not have access to underlying table,view is built on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NE9HeC5WHbjDMYPMr6FrOQ.png" /><figcaption>Subscriber is not able to query the view shared in shared dataset by publisher</figcaption></figure><p>This might not be an accurate approach to provide subscriber with an additional access to parent dataset.</p><p>To address the problem of accessing views having underlying tables in different dataset(than shared dataset),<strong>Authorise the shared dataset</strong> from parent dataset</p><p><em>An </em><strong><em>authorized dataset</em></strong><em> lets you authorize all of the views in a specified dataset to access the data in a second dataset. With an authorized dataset, you don’t need to configure individual authorized views</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J4Y97veHVSY80rJA4-rDJg.png" /><figcaption><strong>Authorising shared dataset from Parent dataset(test_csv) containing table referred in view in shared dataset</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QeKoBz5jpWx5qRBAB0DgvQ.png" /><figcaption><strong>Publisher added Authorisation for Shared dataset</strong></figcaption></figure><h4><strong>Share External Tables as Biglake Tables</strong></h4><p>Sharing an external table provided in linked dataset by Publisher, Subscriber might not be able to query/view external table data due to restricted access to underlying GCS bucket object</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9eqo1GAYp73m8N1IRoVs8Q.png" /><figcaption>Subscriber is not able to query the external table provided in shared dataset by publisher</figcaption></figure><p>So the recommended approach would be Sharing External Table as <strong>Biglake table.</strong>This gives access to Subscriber to query external table without any requirement of providing an extra access to Subscriber for GCS bucket object.</p><p><strong><em>BigLake</em></strong><em> </em><strong><em>tables</em></strong><em> lets you to query structured data in external data stores with access delegation. Access delegation decouples access to the BigLake table from access to the underlying data store.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZLYuNrpDpNC3LdytEiznA.png" /><figcaption>Publisher creating Biglake table to fetch external data source i.e. GCS object</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUxjqzLXqZ1osgcUK_H12g.png" /><figcaption>Subscriber accessing Biglake table without requiring access to underlying GCS object(LHS).On the other hand Subscriber is not able to access data in normal external table without access on GCS bucket(RHS)</figcaption></figure><h4><strong>Column Level Filtering</strong></h4><p>As a recommended practice, to restrict sharing of sensitive columns with subscriber, you can implement <strong>column level filtering</strong> on a shared table using <strong>Policy Tags.</strong></p><p>With Policy tags applied on a source layer table,it allows sharing of a subset of columns &amp; limits the sensitive or not required columns from sharing with subscriber.</p><p><em>BigQuery provides fine-grained access to sensitive columns using policy tags, or type-based classification, of data. Using BigQuery column-level access control, you can create policies that check, at query time, whether a user has proper access.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dj7hiPGAIO1itZwY" /><figcaption>Policy Tags applied on selective columns in a shared dataset table</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LCfcU__ZVdf_xZpJGqhqfg.png" /><figcaption>Subscriber is not able to access any of the policy tag applied columns in linked dataset.</figcaption></figure><p>But point to be noted here is,Subscriber will not be able to access any of the policy tag applied columns in linked dataset unless granted Fine Grained Reader / Masked Reader Role on any of tagged columns</p><p><strong>Note:Publisher needs to provide Fine Grained Reader / Masked Reader</strong> <strong>Role individually to subscriber on required(low/medium sensitive data) policy tag applied columns</strong></p><h4><strong>Row Level Filtering</strong></h4><p>To limit sharing of a whole set of records with subscriber, a good practice would be implementation of <strong>row-level access policies</strong>.This ensures filtering out the records not relevant for subscriber.</p><p><em>Row-level security lets you filter data and enables access to specific rows in a table based on qualifying user conditions.The row-level access policies act as filters to hide or display certain rows of data, depending on whether a user or group is in an allowed list.</em></p><p><strong>Note:</strong> <strong>Row level access policies need to be implicitly created for Subscriber &amp; are not inherited if only created for publisher</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ujWmFSfbS-_7mVJtrVFxug.png" /><figcaption>Row level Access Policy created for Publisher providing access to only records for state=’New York’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G141hkIgKsu89xcH-95tSA.png" /><figcaption>Subscriber is not able to access data in the same table in shared dataset</figcaption></figure><p>Row Level access policy created for Subscriber</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdz_xfUnkeY1sU-nHdDIoA.png" /><figcaption>Row level access policy created implicitly for Subscriber</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aWyd_cF8kZPyRX9Bhz2T-g.png" /><figcaption>Only rows having data for ‘New York’ state are accessible by user/subscriber now</figcaption></figure><p>In conclusion, when sharing data on Analytics Hub, it is crucial to follow these best practices of sharing data for different types of bigquery objects.</p><p>Additionally, it is important to consider factors such as permission &amp; roles settings and sharing views in order to ensure that subscribers are able to access the data they need. By implementing these practices, organizations can ensure that data sharing through Analytics Hub is secure, efficient, and effective!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=378d42719045" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/best-practices-of-data-sharing-on-analytics-hub-378d42719045">Best Practices for Data Sharing On Analytics Hub</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I prepared for GCP PDE certification!!]]></title>
            <link>https://medium.com/@thakurswati9/how-i-prepared-for-gcp-pde-certification-6c26b028f40c?source=rss-4f88bd511453------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c26b028f40c</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[gcp-certification]]></category>
            <dc:creator><![CDATA[Thakurswati]]></dc:creator>
            <pubDate>Fri, 07 Oct 2022 10:55:53 GMT</pubDate>
            <atom:updated>2022-10-07T10:55:53.175Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi There!</p><p>I recently took <strong>GCP Professional data engineer</strong> certification exam &amp; here I am sharing my experience for the exam &amp; how I prepared for it.</p><p>By now if you have decided to take up this exam you must be knowing this exam is primarily focused on <strong>Google cloud’s data engineering services</strong>.</p><p>“<em>A Data Engineer should be able to </em><strong><em>design, build, operationalize, secure, and monitor data processing systems</em></strong><em> with a particular emphasis on security and compliance; scalability and efficiency; reliability and fidelity; and flexibility and portability. A Data Engineer should also be able to leverage, deploy, and continuously train pre-existing machine learning models.</em>”</p><p>The Professional Data Engineer exam assesses your ability to:</p><ul><li>Design data processing systems</li><li>Operationalize machine learning models</li><li>Ensure solution quality</li><li>Build and operationalize data processing systems</li></ul><h4>Major Focus Topics of the exam</h4><ul><li>Services: Bigquery, Dataflow,Pub/Sub,Cloud SQL &amp; Spanner,Bigtable</li><li>Security &amp; Access controls</li><li>Recommended Practices</li><li>ML Models-Training &amp; Retraining,Evaluation,ML services</li></ul><h3>Ok cut to the Preparation Plan then!</h3><p><strong>Set a Timeline</strong>:It took me less than 2 months to prepare for this exam while I had prior experience on working with Google cloud.You can set your timeline accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*oYTf7iI4vDpYo7y_iHKdLg.jpeg" /><figcaption><strong>Decide a date for your exam!</strong></figcaption></figure><h4>I followed this 3 step approach</h4><h4><strong>Step 1: Go through official PDE exam guide</strong></h4><p>Get an idea from the exam guide what topics need to be covered, which services are primary focus for the exam and how you can approach your preparation</p><p>https://cloud.google.com/certification/guides/data-engineer</p><h4><strong>Step 2: Enrolled for an online training course</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILSfjzFzOvo-5-NsYgm0PA.jpeg" /><figcaption><strong>Make Notes!</strong></figcaption></figure><p>It gives you an understanding of the all the topics theoretically &amp; also through hands-on labs.Go through any one online course for PDE exam &amp; follow it religiously before jumping to other ones(if needed!)</p><p>I had taken <strong>Pluralsight</strong> Data engineer course, although Acloud guru course material is highly recommended as I heard from my colleagues.</p><p><strong>Qwiklabs</strong> is a good platform to learn via Hands-on Labs</p><p>Took me around a month to complete whole study material ,although I skipped a few hands on labs</p><p>If you prefer going by Book instead of video content,you can follow <strong>Dan</strong> <strong>Sullivan’s</strong> book for <em>Certified Professional Data Engineer Exam. </em><strong><em>Book</em></strong><em> </em>Comes with Chapter wise Practice questions &amp; 2 Practice Tests at the end</p><h4><strong>Step 3:Practice Tests</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Dm4WzWDyJR9pgcA0YUw6lw.jpeg" /></figure><p>Once I was done with the course/study material I started taking practice tests and sample questions</p><p>Some sites you can refer for Practice &amp; sample tests: <strong>Google’s sample questions,Examtopics Question Bank,Whizlabs Practice tests,Dan Sullivan Practice questions</strong></p><p>Once you are done with these 3 steps keep going back to Step 1-To reflect on which topics need to be revisited</p><p>In which areas you are not performing well in practice tests</p><h4><strong>Other Resources</strong>:Other resources which helped me to get through this exam</h4><h4><strong>Youtube Channels</strong></h4><ol><li><strong>Awesome GCP PDE playlist </strong>— <a href="https://www.youtube.com/watch?v=_IeS1m8r6SY&amp;list=PLIivdWyY5sqLsaG5hNms0D9aZRBE7DHBb">https://www.youtube.com/watch?v=_IeS1m8r6SY&amp;list=PLIivdWyY5sqLsaG5hNms0D9aZRBE7DHBb</a></li></ol><p><strong>2. Learn GCP with Mahesh </strong><a href="https://www.youtube.com/c/LearnGCPwithMahesh/featured">https://www.youtube.com/c/LearnGCPwithMahesh/featured</a></p><p><strong>3. Google Cloud Tech </strong><a href="https://www.youtube.com/user/googlecloudplatform">https://www.youtube.com/user/googlecloudplatform</a></p><p><strong>4. Cloud Bytes Playlist </strong><a href="https://www.youtube.com/watch?v=CFw4peH2UwU&amp;list=PLIivdWyY5sqIQ4_5PwyyXZVdsXr3wYhip">https://www.youtube.com/watch?v=CFw4peH2UwU&amp;list=PLIivdWyY5sqIQ4_5PwyyXZVdsXr3wYhip</a></p><p><strong>5. Cloud Beginner’s playlist by IT k Funde </strong><a href="https://www.youtube.com/watch?v=OzwSBbuHY-0&amp;list=PLcnJIHtHiTA2Jp3klIkeHiwqGrxjXBzh6">https://www.youtube.com/watch?v=OzwSBbuHY-0&amp;list=PLcnJIHtHiTA2Jp3klIkeHiwqGrxjXBzh6</a></p><h4><strong>Book</strong></h4><p><strong>Dan Sullivan Book for Certified Professional Data Engineer Exam</strong></p><p>Alright this is what I followed for my PDE exam preparation.If you know any other essential resources do let me know in the comments :)</p><h3>Happy Learning!!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c26b028f40c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>