<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by CPiramide on Medium]]></title>
        <description><![CDATA[Stories by CPiramide on Medium]]></description>
        <link>https://medium.com/@piramidec?source=rss-29f641a25414------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*D_BjNZJhQpxYekHV.jpg</url>
            <title>Stories by CPiramide on Medium</title>
            <link>https://medium.com/@piramidec?source=rss-29f641a25414------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:35:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@piramidec/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Google Gemini for Automatic Data Cataloging]]></title>
            <link>https://medium.com/@piramidec/google-gemini-for-automatic-data-cataloging-7098f4947a87?source=rss-29f641a25414------2</link>
            <guid isPermaLink="false">https://medium.com/p/7098f4947a87</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-developer-group]]></category>
            <dc:creator><![CDATA[CPiramide]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 13:44:26 GMT</pubDate>
            <atom:updated>2025-04-23T13:44:26.516Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>The importance of cataloging tables and columns in data platforms <strong>cannot be overstated.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*0BqgLWsUaGWJ6TCk.gif" /></figure><p>With the exponential growth of data volumes, organizations are facing significant challenges in <strong>organizing</strong>, <strong>accessing</strong>, and effectively <strong>managing</strong> <strong>their</strong> <strong>data</strong>.<br><strong>Data</strong> <strong>cataloging</strong> <strong>simplifies</strong> these processes by systematically identifying and labeling tables and columns, ensuring that data is easily found and accessible. By leveraging advanced technologies such as Google’s GenAI (Gemini), organizations can significantly increase operational efficiency, scalability, and data governance, ultimately transforming their data management practices.<br>By introducing an automatic cataloging process, time-consuming and error-prone tasks are reduced, allowing data consumers to focus on strategic analysis and insight extraction.<strong> The final check is still in the hands of humans, who now have the role of validator of the cataloging obtained.</strong></p><h3><strong>Set-up</strong></h3><p><strong>Metadata-wizard</strong></p><p>this is the tool available in the project named dataplexutils in the repo named GoogleCloudPlatform on github. official: <a href="https://github.com/GoogleCloudPlatform/dataplexutils">https://github.com/GoogleCloudPlatform/dataplexutils</a></p><p><strong>Example table</strong></p><p>We create an ad hoc table for this example.</p><p>Table schema:</p><pre>CREATE TABLE luxottica_claudio.customer_information<br>(<br>    -- Customer basic information<br>    customer_id INT64,<br>    first_name STRING,<br>    last_name STRING,<br>    email STRING,<br>    phone_number STRING,<br>    <br>    -- Location information<br>    street_address STRING,<br>    city STRING,<br>    state STRING,<br>    postal_code STRING,<br>    country STRING,<br>    <br>    -- Last transaction details<br>    last_transaction_date TIMESTAMP,<br>    last_transaction_amount NUMERIC,<br>    last_transaction_id STRING,<br>    <br>    -- Metadata<br>    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP(),<br>    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP()<br>)<br>PARTITION BY DATE(last_transaction_date)<br>CLUSTER BY country, state, customer_id;</pre><p>Use Faker to populate the table</p><pre>from faker import Faker<br>from datetime import datetime, timedelta<br>import random<br>import decimal<br>from google.cloud import bigquery<br>from google.api_core import retry<br><br># Initialize Faker and BigQuery client<br>fake = Faker()<br>PROJECT_ID = &quot;&lt;INSERT THE PROJECT ID&gt;&quot;<br>client = bigquery.Client(project=PROJECT_ID)<br><br>def generate_customer_data(num_records=10000):<br>    rows_to_insert = []<br>    <br>    for _ in range(num_records):<br>        # Generate basic customer information<br>        row = {<br>            &#39;customer_id&#39;: _ + 1,<br>            &#39;first_name&#39;: fake.first_name(),<br>            &#39;last_name&#39;: fake.last_name(),<br>            &#39;email&#39;: fake.email(),<br>            &#39;phone_number&#39;: fake.phone_number(),<br>            <br>            # Location information<br>            &#39;street_address&#39;: fake.street_address(),<br>            &#39;city&#39;: fake.city(),<br>            &#39;state&#39;: fake.state(),<br>            &#39;postal_code&#39;: fake.postcode(),<br>            &#39;country&#39;: fake.country(),<br>            <br>            # Transaction details<br>            &#39;last_transaction_date&#39;: fake.date_time_between(<br>                start_date=&#39;-1y&#39;,<br>                end_date=&#39;now&#39;<br>            ),<br>            &#39;last_transaction_amount&#39;: decimal.Decimal(<br>                round(random.uniform(10.0, 1000.0), 0)<br>            ),<br>            &#39;last_transaction_id&#39;: fake.uuid4()<br>        }<br>        <br>        rows_to_insert.append(row)<br>    <br>    return rows_to_insert<br><br>def insert_to_bigquery(rows):<br>    # Update these with your actual project and dataset IDs<br>    table_id = &quot;&lt;PROJECT_ID&gt;.&lt;DATASET&gt;.&gt;.&lt;TABLE&gt;&quot;<br>    <br>    # Get table reference<br>    table = client.get_table(table_id)<br>    <br>    # Insert rows with retry mechanism<br>    errors = client.insert_rows(<br>        table, <br>        rows,<br>        retry=retry.Retry(deadline=30)<br>    )<br>    <br>    if errors == []:<br>        print(f&quot;Successfully inserted {len(rows)} rows&quot;)<br>    else:<br>        print(&quot;Encountered errors while inserting rows:&quot;, errors)<br><br>if __name__ == &quot;__main__&quot;:<br>    # Generate the data<br>    print(&quot;Generating data...&quot;)<br>    rows = generate_customer_data(10000)<br>    <br>    # Insert into BigQuery<br>    print(&quot;Inserting data into BigQuery...&quot;)<br>    insert_to_bigquery(rows)</pre><p>Once we have populated the table, connecting to the BigQuery console we see something like that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gEK5MVdfi91ObMCMgqctaw.png" /></figure><p>We can see that the table columns has no associated <strong>Description </strong>and also the table has no description associated</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_1sBPzRB327uC6P9tWtxHw.png" /></figure><h3>Usage of metadata-wizard</h3><p>I created a notebook in order to execute some test on the library metadata-wizard.</p><p>Lets start to initialize some variables:</p><pre>import sys<br>import os<br>sys.path.append(&#39;PATH TO dataplexutils/src/package&#39;)<br>from dataplexutils.metadata.wizard import Client, ClientOptions<br><br>os.environ[&#39;GOOGLE_CLOUD_PROJECT&#39;] = &#39;&lt;PROJECT_ID&gt;&#39;<br>PROJECT_ID = &quot;&lt;PROJECT_ID&gt;&quot;<br>DATASET_ID = &quot;&lt;DATASET_ID&gt;&quot;<br>TABLE_ID = &quot;&lt;TABLE_ID&gt;&quot;<br>LLM_LOCATION = &quot;us-central1&quot; #or your preferite zone<br>DATAPLEX_LOCATION =  &quot;us-central1&quot; #or your preferite zone<br>DOCUMENTATION_URI = &quot;us-central1&quot; #or your preferite zone<br><br>table_fqn = f&quot;{PROJECT_ID}.{DATASET_ID}.{TABLE_ID}&quot;</pre><p>Initialize the Client Options object with some parameters to configure the tool behaviour. in this example I’ll not provide external PDF to help the GenAI engine to understand the context.</p><pre>client_options = ClientOptions(<br>    use_lineage_tables=False, #this will take into account lineage information if available<br>    use_lineage_processes=False, #this will take into account lineage information if available<br>    use_profile=True, #this will take into account table profiling information<br>    use_data_quality=True, #this will take into account table data quality scaninformation<br>    use_ext_documents=False #this will take into account external document<br>)<br><br>client = Client(<br>    project_id=PROJECT_ID,<br>    llm_location=LLM_LOCATION,<br>    dataplex_location=DATAPLEX_LOCATION,<br>    client_options=client_options<br>)</pre><p>finally</p><pre>client.generate_table_description(table_fqn)</pre><p>this operation will generate for the identified table the description. as result we can see in the description of the table that now the description is populated and reflect the content of that table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bvjD4mcct2ck9YbEZuU4MQ.png" /></figure><p>and with</p><pre>client.generate_columns_descriptions(table_fqn)</pre><p>each column now has the related description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vxbKyxlGUc1guIvaloqXxg.png" /></figure><h3><strong>Conclusion</strong></h3><p>This tool certainly <strong>helps to start a cataloging process</strong> of the data present in a data platform. I do not think that the role of the data steward can be eliminated completely, it is certainly facilitated by the adoption of tools like this.<br>The tool is completely configurable (e.g. you can change the prompt sent to the LLM, select a specific LLM etc.)</p><h3>References and Additional Resources</h3><h4>Notebook:</h4><pre># %% [markdown]<br># Copyright 2024 Google LLC<br># <br># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);<br># you may not use this file except in compliance with the License.<br># You may obtain a copy of the License at<br># <br>#     https://www.apache.org/licenses/LICENSE-2.0<br># <br># Unless required by applicable law or agreed to in writing, software<br># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,<br># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.<br># See the License for the specific language governing permissions and<br># limitations under the License.<br><br># %% [markdown]<br># # dataplex-utils<br># ## metadata<br># ### wizard<br><br># %% [markdown]<br># Build the package launching `source build_install_package.sh` from Terminal<br><br># %% [markdown]<br># Restart notebook kernel<br><br># %%<br><br>import sys<br>import os<br>sys.path.append(&#39;PATH TO dataplexutils/src/package&#39;)<br><br>from dataplexutils.metadata.wizard import Client, ClientOptions<br><br>os.environ[&#39;GOOGLE_CLOUD_PROJECT&#39;] = &#39;&lt;PROJECT_ID&gt;&#39;<br>PROJECT_ID = &quot;&lt;PROJECT_ID&gt;&quot;<br>DATASET_ID = &quot;&lt;DATASET_ID&gt;&quot;<br>TABLE_ID = &quot;&lt;TABLE_ID&gt;&quot;<br><br>LLM_LOCATION = &quot;us-central1&quot;<br>DATAPLEX_LOCATION =  &quot;us-central1&quot;<br>DOCUMENTATION_URI = &quot;us-central1&quot;<br><br># %%<br>client_options = ClientOptions(<br>    use_lineage_tables=False,<br>    use_lineage_processes=False,<br>    use_profile=True,<br>    use_data_quality=True,<br>    use_ext_documents=False<br>    )<br><br>client = Client(<br>    project_id=PROJECT_ID,<br>    llm_location=LLM_LOCATION,<br>    dataplex_location=DATAPLEX_LOCATION,<br>    client_options=client_options<br>)<br><br># %%<br>table_fqn = f&quot;{PROJECT_ID}.{DATASET_ID}.{TABLE_ID}&quot;<br>dataset_fqn = f&quot;{PROJECT_ID}.{DATASET_ID}&quot;<br><br><br># %%<br>client.generate_table_description(table_fqn)<br><br># %%<br>client.generate_columns_descriptions(table_fqn)</pre><h4>Metadata-wizard:</h4><p><a href="https://github.com/GoogleCloudPlatform/dataplexutils">https://github.com/GoogleCloudPlatform/dataplexutils</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7098f4947a87" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>