<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prasanna Kumar on Medium]]></title>
        <description><![CDATA[Stories by Prasanna Kumar on Medium]]></description>
        <link>https://medium.com/@prasy77?source=rss-f847b7322cb9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sfHpjTKYhjh5zMyeqQ3XlA.jpeg</url>
            <title>Stories by Prasanna Kumar on Medium</title>
            <link>https://medium.com/@prasy77?source=rss-f847b7322cb9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:49:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prasy77/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Database roles in snowflake]]></title>
            <link>https://medium.com/@prasy77/database-roles-in-snowflake-8e2f7f2751f0?source=rss-f847b7322cb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e2f7f2751f0</guid>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[database-role]]></category>
            <dc:creator><![CDATA[Prasanna Kumar]]></dc:creator>
            <pubDate>Thu, 12 Oct 2023 20:24:32 GMT</pubDate>
            <atom:updated>2023-10-13T10:06:17.483Z</atom:updated>
            <content:encoded><![CDATA[<p>Snowflake has introduced database roles concept, facilitating the assignment of grants on database objects to other database roles or snowflake roles. This simplifies the process of redistributing database roles to multiple roles without ambiguity.</p><p>Below is a simple illustration of a database role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*zkSct_D2jA41tnM8tFdcSw.png" /></figure><p>Pre-requisites</p><p>Trail snowflake account:</p><p>Use the link <a href="https://signup.snowflake.com/">Snowflake Trial</a> to create a snowflake trail account.</p><p>Once snowflake account is created follow the below queries.</p><pre>--create database<br>create database DB_CITIES;</pre><pre>--create schema<br>create schema CITIES;</pre><pre>--create table<br>create table  CITIES_LIST (CITIES_NAME  VARCHAR);</pre><pre>--Insert cities<br>INSERT INTO CITIES_LIST(CITIES_NAME) VALUES (&#39;MANCHESTER&#39;);<br>INSERT INTO CITIES_LIST(CITIES_NAME) VALUES (&#39;BRIGHTON&#39;);<br>INSERT INTO CITIES_LIST(CITIES_NAME) VALUES (&#39;LONDON&#39;);</pre><pre>--create DB role for cities access<br><br>create or replace database role CITIES_DB_READER;</pre><pre>--Grants on database objects to database role <br>GRANT USAGE on schema DB_CITIES.CITIES to DATABASE ROLE CITIES_DB_READER;<br>GRANT SELECT on ALL TABLES IN DATABASE DB_CITIES TO DATABASE ROLE CITIES_DB_READER;</pre><pre><br>--show grants to verify the correctness of above steps<br><br>show grants to database role CITIES_DB_READER;</pre><pre>--create snowflake role<br>create ROLE CITIES_READER;<br>--grant database role to snowflake role<br>GRANT DATABASE ROLE CITIES_DB_READER to ROLE CITIES_READER;</pre><pre>--create user<br>create or replace user CITIES_ANALYST password = &#39;&lt;as-per-user&gt;&#39; must_change_password = false;</pre><pre>--grant role to use compute_wh<br>grant modify,monitor,operate,usage on warehouse COMPUTE_WH to role CITIES_READER;</pre><pre>-- grant role cities_reader to cities_analyst user<br>grant role CITIES_READER to user CITIES_ANALYST;</pre><p>After using this series of steps, the database role will be granted to snowflake role i,e .. further granted to user..</p><p>Now login to the snowflake account with credentials of CITIES_ANALYST.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nmpwJuWCXs0SiZ7oMM-Jw.png" /></figure><p>Here the CITIES_ANALYST user is mapped to CITIES_READER role is shown.</p><p>Now query the CITIES_LIST table and observe the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYGn2YOZf6_ufC43QVqLsQ.png" /></figure><p>This feature can be used to implement complex role management to support concepts like datamesh as well.</p><blockquote>official documentation</blockquote><blockquote><a href="https://docs.snowflake.com/en/sql-reference/snowflake-db-roles">SNOWFLAKE Database Roles | Snowflake Documentation</a></blockquote><p>Follow me for more content : <a href="https://medium.com/@prasy77">Prasanna Kumar — Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e2f7f2751f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Local Containerized DBT development with VSCODE Snowflake]]></title>
            <link>https://towardsdev.com/local-containerized-dbt-development-with-vscode-snowflake-378cfa13a956?source=rss-f847b7322cb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/378cfa13a956</guid>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Prasanna Kumar]]></dc:creator>
            <pubDate>Sat, 27 May 2023 11:43:15 GMT</pubDate>
            <atom:updated>2023-07-19T07:01:22.713Z</atom:updated>
            <content:encoded><![CDATA[<p>In this setup, the focus is on using Docker containers, Visual Studio Code (VSCode), and Snowflake to facilitate the development of DBT (Data Build Tool) projects.</p><p>By containerizing the DBT development environment, developers can achieve consistency and portability across different machines and environments. The use of Docker containers allows for easy setup and replication of the development environment, including all the necessary dependencies and configurations.</p><p><strong>Introduction to tools</strong></p><p><strong>DBT: </strong>DBT is a popular open-source tool used for transforming and modeling data in a structured, testable, and version-controlled manner.</p><p><strong>VSCODE: </strong>Visual Studio Code (VSCode) is a lightweight and feature-rich code editor that provides a seamless development experience for various programming languages, including SQL and Python(many other programming languages), which are commonly used with DBT.</p><p><strong>Snowflake:</strong> Snowflake is a cloud-based data warehousing platform known for its scalability, performance, and ease of use. It provides a powerful environment for storing, querying, and analyzing data.</p><p>create snowflake trail account using the below link.</p><p><a href="https://signup.snowflake.com/"><em>https://signup.snowflake.com</em></a></p><p>Steps involved:</p><p>Create a new directory to house your DBT projects using the command: mkdir dbt-projects.Navigate to the newly created directory with: cd dbt-projects.You are now ready to start developing your DBT projects within the dbt-projects directory.</p><pre>mkdir dbt-projects<br>cd dbt-projects</pre><p>Create a Python virtual environment for DBT projects using python3 -m venv .pydbt and initialize it as .pydbt.</p><pre>python3 -m venv .pydbt</pre><p>Activate the Python virtual environment for DBT projects by running the command source .pydbt/bin/activate.</p><pre>source .pydbt/bin/activate</pre><p>Install the dbt-snowflake package within the activated Python virtual environment for DBT projects using the command pip install dbt-snowflake.</p><pre>pip install dbt-snowflake</pre><p>Verify the successful installation of DBT by checking the version using the command dbt --version within the activated Python virtual environment for DBT projects.</p><pre>dbt --version<br></pre><p>Create a folder named `sample-dbt` and navigate into it by executing the command `mkdir sample-dbt &amp;&amp; cd sample-dbt`.This is for creating dbt template project with dbt init</p><pre>mkdir sample-dbt &amp;&amp; cd sample-dbt</pre><p>create dbt-sample project using dbt init and follow the instructions as showed below.</p><pre>(.pydbt) ➜  sample-dbt dbt init<br>09:55:37  Running with dbt=1.5.0<br>Enter a name for your project (letters, digits, underscore): dbt_sample<br>Which database would you like to use?<br>[1] snowflake<br><br>(Don&#39;t see the one you want? https://docs.getdbt.com/docs/available-adapters)<br><br>Enter a number: 1<br>account (https://&lt;this_value&gt;.snowflakecomputing.com): &lt;Accountid&gt;.eu-north-1.aws<br>user (dev username): &lt;username&gt;<br>[1] password<br>[2] keypair<br>[3] sso<br>Desired authentication type option (enter a number): 1<br>password (dev password): <br>role (dev role): ACCOUNTADMIN<br>warehouse (warehouse name): COMPUTE_WH<br>database (default database that dbt will build objects in): &lt;databasename&gt;<br>schema (default schema that dbt will build objects in): &lt;schema&gt;<br>threads (1 or more) [1]: 16<br>09:57:11  Profile dbt_sample written to /Users/*****/.dbt/profiles.yml using target&#39;s profile_template.yml and your supplied values. Run &#39;dbt debug&#39; to validate the connection.<br>09:57:11  <br>Your new dbt project &quot;dbt_sample&quot; was created!<br><br>For more information on how to configure the profiles.yml file,<br>please consult the dbt documentation here:<br><br>  https://docs.getdbt.com/docs/configure-your-profile<br><br>One more thing:<br><br>Need help? Don&#39;t hesitate to reach out to us via GitHub issues or on Slack:<br><br>  https://community.getdbt.com/<br><br>Happy modeling!</pre><p>Please find the below project structure in VSCODE.Your dbt template is ready and also verified the connection from dbt to snowflake using <strong><em>“dbt debug” </em></strong>Command</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l82VReMPJDtk0fgmnNoYdQ.png" /></figure><p>Go to the parent folder i,e sample-dbt where dbt_sample project reside and create the docker file with naming convention <strong>Dockerfile </strong>with the below content</p><pre># Use the Python 3.10 slim-buster base image<br>FROM python:3.10-slim-buster<br><br># Update package lists and install necessary dependencies<br>RUN apt-get update \<br>    &amp;&amp; apt-get install -y --no-install-recommends<br><br># Set the working directory within the container<br>WORKDIR /usr/src/dbt/sample-dbt<br><br># Create a volume for /usr/src<br>VOLUME [ &quot;/usr/src&quot; ]<br><br># Upgrade pip to the latest version<br>RUN pip install --upgrade pip<br><br># Print the current working directory<br>RUN pwd<br><br># Install the dbt Snowflake adapter and dbt-core<br>RUN pip install dbt-snowflake<br><br># Create a directory for dbt configuration files<br>RUN cd /root &amp;&amp; mkdir /root/.dbt<br><br># Copy the profiles.yml file to the /root/.dbt directory<br>COPY profiles/profiles.yml /root/.dbt<br><br># Set the entrypoint to &quot;dbt&quot;<br>ENTRYPOINT [ &quot;dbt&quot; ]</pre><p>Now the directory structure of sample-dbt must be like as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*YNGDVcDZ2TnrYz0xScQloA.png" /></figure><p>Add profile information as profiles.yml to profiles folder under sample-dbt to copy to container.Now the structure of the project should be like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*xcK3n-W4QjErWGsSk9ST9Q.png" /></figure><p>Use ctrl+shift+p( windows) or cmd+shift+p (mac) to get to the below image where it gives options how to add development container configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3hGN6yyHSr4j73M2fKyLiA.png" /></figure><p>Use existing docker file option to build the Configuration and open the bash terminal to do the development of DBT in container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDbA8rkq6CS1vhObKUxzmQ.png" /></figure><p>In this setup, you’ll benefit from the following key features:</p><ol><li>Consistent and Isolated Development Environment: Docker containers provide a consistent and isolated environment for DBT project development, ensuring that dependencies, libraries, and configurations remain intact regardless of the underlying system.</li><li>Reproducible Builds: Defining the DBT development environment as a Docker image facilitates easy replication across different machines.</li></ol><p><strong>References:</strong> <a href="https://towardsdatascience.com/create-local-dbt-project-e12c31bd3992">https://towardsdatascience.com/create-local-dbt-project-e12c31bd3992</a></p><p>Please follow me for more content and grow together</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=378cfa13a956" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/local-containerized-dbt-development-with-vscode-snowflake-378cfa13a956">Local Containerized DBT development with VSCODE Snowflake</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Copying files from Azure Datalake to Google Storage Bucket]]></title>
            <link>https://medium.com/@prasy77/copying-files-from-azure-datalake-to-google-storage-bucket-9e396e958800?source=rss-f847b7322cb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e396e958800</guid>
            <category><![CDATA[azure-data-lake]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud-storage]]></category>
            <category><![CDATA[google-storage]]></category>
            <dc:creator><![CDATA[Prasanna Kumar]]></dc:creator>
            <pubDate>Tue, 04 Jan 2022 20:19:43 GMT</pubDate>
            <atom:updated>2022-01-04T20:19:43.322Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I have been exploring options for transferring files from Azure Datalake to GCS.I know there are many ways to do it.But I want something which is cost effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*ROYs0MaZkKqhJaFXA-K84g.jpeg" /><figcaption><strong>Azure datalake to GCP storage Adhoc datacopy</strong></figcaption></figure><p>Google storage transfer is the preferred way to copy the data from Azure blob storage to Google Cloud Storage.</p><p>Other way is using the combination of AZCOPY and GCSFUSE to copy the files from blob storage to compute machine running on GCP and mount the files to the Google Cloud Storage.</p><p>You can create the VM based on the size of the files you like to download from Azure storage and mount them on the GCP storage</p><p>Azure documentation explained how to use AZCOPY and use the below links to get the understanding over it.</p><blockquote><a href="https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy#:~:text=AzCopy%20is%20a%20command-line%20tool%20that%20moves%20data,can%20provide%20authorization%20credentials%20to%20the%20storage%20service.">azcopy | Microsoft Docs</a></blockquote><p><strong>Sequence of Steps:</strong></p><p><em>Create the compute engine of Linux OS</em></p><p><em>SSH into the Compute engine and run the following commands to make your Linux environment (open the VM in a browser)</em></p><p><em>Update your apt-get package lists by entering the following command in the terminal:</em></p><blockquote><strong>sudo apt-get update</strong></blockquote><p><em>Install wget by entering the following command:</em></p><blockquote><strong><em>sudo apt-get install wget</em></strong></blockquote><p><em>Verify that wget is active</em></p><blockquote><strong>wget — version</strong></blockquote><p><em>create a new directory where you want to copy files from multiple azure folders</em></p><p><em>Download AzCopy</em></p><blockquote><strong>wget </strong><a href="https://aka.ms/downloadazcopy-v10-linux"><strong>https://aka.ms/downloadazcopy-v10-linux</strong></a></blockquote><p><em>Extract from Archive</em></p><blockquote><strong>tar -xvf downloadazcopy-v10-linux</strong></blockquote><p><em>Move to bin</em></p><blockquote><strong>sudo cp ./azcopy_linux_amd64_*/azcopy /usr/bin/</strong></blockquote><p>Now AZCopy installation is completed.</p><p><strong>Now the GCS bucket has to be mounted on the directory in the virtual machine using GCS Fuse.</strong></p><p>Use the below</p><blockquote><strong>export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`<br>echo “deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main” | sudo tee /etc/apt/sources.list.d/gcsfuse.list<br>curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -<br>sudo apt-get update -y<br>sudo apt-get install gcsfuse -y</strong></blockquote><p>Now create two directories one for downloading all the files and if any necessary small cosmetics changes to the filenames or gathering files from multiple process.</p><blockquote><strong>mkdir tempdir</strong></blockquote><blockquote><a href="https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2019-02-02&amp;st=2019-04-29T22%3A18%3A26Z&amp;se=2019-04-30T02%3A23%3A26Z&amp;sr=b&amp;sp=rw&amp;sip=168.1.5.60-168.1.5.70&amp;spr=https&amp;sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D"><strong>https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt</strong>?sv=2019-02-02&amp;st=2019-04-29T22%3A18%3A26Z&amp;se=2019-04-30T02%3A23%3A26Z&amp;sr=b&amp;sp=rw&amp;sip=168.1.5.60-168.1.5.70&amp;spr=https&amp;sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D</a></blockquote><p>you can differentiate the path and SAS in the above.SAS can be generated via console.</p><blockquote><strong>azcopy cp </strong><a href="https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2019-02-02&amp;st=2019-04-29T22%3A18%3A26Z&amp;se=2019-04-30T02%3A23%3A26Z&amp;sr=b&amp;sp=rw&amp;sip=168.1.5.60-168.1.5.70&amp;spr=https&amp;sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D"><strong>https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?[</strong></a><strong>SAS] [tempdir]</strong></blockquote><p><em>Repeat the same process if you have other files to copy to same bucket.</em></p><p><em>Now create a directory and mount it with GCS bucket</em></p><blockquote><strong>mkdir finaldir</strong></blockquote><p><em>Mount GCS bucket</em></p><blockquote><strong>gcsfuse &lt;gcpbucket&gt; &lt;finaldir&gt;</strong></blockquote><p><em>Move all the files from tempdir to final dir</em></p><blockquote><strong>mv tempdir/* finaldir/*</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e396e958800" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authenticating and Interacting with Google Drive]]></title>
            <link>https://medium.com/@prasy77/authenticating-and-interacting-with-google-drive-ec843330d109?source=rss-f847b7322cb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec843330d109</guid>
            <category><![CDATA[google-drive-api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[google-technologies]]></category>
            <category><![CDATA[google-developer-group]]></category>
            <category><![CDATA[google-api]]></category>
            <dc:creator><![CDATA[Prasanna Kumar]]></dc:creator>
            <pubDate>Wed, 24 Nov 2021 09:33:37 GMT</pubDate>
            <atom:updated>2021-11-24T09:33:37.558Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*1kAin8rCrvbdy1WDfovKdA.png" /></figure><p>Google exposed API’s to the developers of this world for accessing the required information with fair usage and minimum risk.To access any Google API the below are few things to consider.</p><ol><li>Access scope required for accessing the data</li><li>Required Authentication details</li><li>How to use the refresh token?</li></ol><p>All the information required for this is already published in Google developers website but I want to explain it organically so that it is easy to understand.</p><p>Prerequisites</p><p>Google Cloud Account</p><p>Basic python programming</p><p>Google Drive + Account details</p><p>Setting up OAUTH authentication is the main and crucial step which needs to be understood clearly.Below are the steps given for setting up OAUTH authentication.</p><p><strong><em>Google Drive</em></strong></p><p>As a first step to access google drive,please make sure you have access to google drive and its corresponding google account.I have google drive and google account with same google account.</p><p><strong><em>Enabling API</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5v1d3iwoF6yRmzBYl4-uhA.png" /></figure><p>Search for Google drive API in Google cloud console and enable API.</p><p><strong><em>Setting up OAUTH consent</em></strong></p><p>Create an app for OAUTH consent which is later used to give the consent of the user to access the particular application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EDUeoEzUTYkGTgcW02FP9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*Njbe1gCtz7T924WfQIPhkA.png" /></figure><p>Use add and remove scopes to add required auth scope for accessing google drive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*SDCbRZB_18MnaHqK9_J81Q.png" /></figure><p>Here I have created an app for OAUTH consent in order to access app now the next step is to create the credentials to access this consent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*WqM4xB35eDfIMtu1sKAm8A.png" /></figure><p>select OAUTH client ID as the option while creating the credentials and download the json file and save it as credentials.json</p><p>Important python libraries to install in the python environment.</p><pre><em>pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib</em></pre><blockquote>#Available from Google Developers quickstart samples<br><em>from __future__ import print_function<br>import os.path<br>from googleapiclient.discovery import build<br>from google_auth_oauthlib.flow import InstalledAppFlow<br>from google.auth.transport.requests import Request<br>from google.oauth2.credentials import Credentials<br>import google.oauth2.credentials<br>import google_auth_oauthlib.flow</em></blockquote><blockquote><em># If modifying these scopes, delete the file token.json.<br>SCOPES = [‘</em><a href="https://www.googleapis.com/auth/drive.metadata.readonly&#39;"><em>https://www.googleapis.com/auth/drive.metadata.readonly&#39;</em></a><em>]</em></blockquote><blockquote><em>def main():<br> “””Shows basic usage of the Drive v3 API.<br> Prints the names and ids of the first 10 files the user has access to.<br> “””<br> creds = None<br> # The file token.json stores the user’s access and refresh tokens, and is<br> # created automatically when the authorization flow completes for the first<br> # time.<br> #this file will be created on fly<br> creds_path=’token.json’<br> if os.path.exists(creds_path):<br> creds = Credentials.from_authorized_user_file(creds_path, SCOPES)<br> # If there are no (valid) credentials available, let the user log in.<br> if not creds or not creds.valid:<br> if creds and creds.expired and creds.refresh_token:<br> creds.refresh(Request())<br> else:<br> flow = InstalledAppFlow.from_client_secrets_file(<br> r’&lt;credentials_file_path&gt;/credentials.json’, SCOPES)<br> creds = flow.run_local_server(port=0)<br> # Save the credentials for the next run<br> with open(‘token.json’, ‘w’) as token:<br> token.write(creds.to_json())</em></blockquote><blockquote><em>service = build(‘drive’, ‘v3’, credentials=creds)</em></blockquote><blockquote><em># Call the Drive v3 API<br> results = service.files().list(<br> pageSize=10, fields=”nextPageToken, files(id, name)”).execute()<br> items = results.get(‘files’, [])</em></blockquote><blockquote><em>if not items:<br> print(‘No files found.’)<br> else:<br> print(‘Files:’)<br> for item in items:<br> print(u’{0} ({1})’.format(item[‘name’], item[‘id’]))</em></blockquote><blockquote><em>if __name__ == ‘__main__’:<br> main()</em></blockquote><p>Here token.json creates on runtime and the refresh token present in the json file is used to access the application.</p><p>For the first time one has to manually authenticate to create the file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qs_-WCElP7AeLZIYof3-mw.png" /></figure><p>This is the OAUTH authentication consent which will create refresh token and store in JSON file.</p><p>This is just a sample program how can we authenticate to google drive and several functionalities can be achieved through this.</p><p>Please follow <a href="https://medium.com/@prasy77">Prasanna Kumar — Medium</a> for more stories.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec843330d109" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pub/Sub Notifications via GCS triggering Cloud function to perform a micro activity]]></title>
            <link>https://medium.com/@prasy77/pub-sub-notifications-triggering-cloud-function-to-perform-a-micro-activity-45d2fe9ab087?source=rss-f847b7322cb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/45d2fe9ab087</guid>
            <category><![CDATA[google-cloud-storage]]></category>
            <category><![CDATA[pub-sub]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud-functions]]></category>
            <dc:creator><![CDATA[Prasanna Kumar]]></dc:creator>
            <pubDate>Sat, 30 Oct 2021 16:48:37 GMT</pubDate>
            <atom:updated>2021-10-30T22:01:22.420Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>How to pass values present in GCS file to Pubsub to triggering a cloud function for certain specific activity.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v3sB1PjBGQyBi1vtfu4s4Q.jpeg" /><figcaption>PubSub Interactions with GCS</figcaption></figure><p>I am in quest of answering some of the business requirements through Google Cloud Platform.The challenge is to pass arguments to any business specific functionality dynamically and recursively from a file in a GCS storage.</p><p>This idea will help us in triggering a cloud function with a specific functionality receiving different parameters.</p><p><strong>Prerequisites:</strong></p><p>Google cloud platform subscription,Python basic programming,Enable required Google services API’s</p><p><strong>Google Storage:</strong></p><p>I have created a file in Google storage bucket with a small CSV file with 3 records</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*5_mVWDb9vKeTXTVFHqas5Q.png" /><figcaption>CSV delimited file</figcaption></figure><p><strong>Create a Publisher Topic</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UjPGMv1PwtoAFNvKjcIoJg.png" /><figcaption>Pubsub Topic creation</figcaption></figure><p><strong>Create Subscriber</strong></p><p>Subscriber gets automatically created when you create Publisher if not create Subscriber similar to Publisher</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fhpd0Ak1QT-6UuIDIzhtEA.png" /><figcaption>Subscriber</figcaption></figure><p><strong>Create Service account and give owner access to the project</strong></p><p>The service account JSON file is used to access the GCP services.</p><p>I have the below python code to read the contents of the GCS file and pass that a messages to the Publisher</p><blockquote>import os<br>from io import StringIO<br>import time<br>from google.cloud import pubsub_v1 #importing Pubsub<br>from google.cloud import storage #importing storage<br>publisher=pubsub_v1.PublisherClient.from_service_account_json(r’&lt;Serviceaccountfilename&gt;’)<br>storage_client = storage.Client.from_service_account_json(r’&lt;Serviceaccountfilename&gt;’)<br>topic_path=’&lt;PubsubTopicname&gt;’ #topic name<br>bucket = storage_client.get_bucket(‘&lt;BucketName&gt;’) #bucket name<br>blob = bucket.blob(‘&lt;CSVFilename&gt;’) #file in blob storage<br>blob = blob.download_as_string()<br>blob = blob.decode(‘utf-8’)<br>blob = StringIO(blob) #tranform bytes to string here<br>names = csv.reader(blob) #then use csv library to read the content<br>for name in names:<br> data = name[0].split(‘;’)[1]<br> data = data.encode(‘utf-8’)<br> future = publisher.publish(topic_path,data) #publishing message to Pubsub<br> print(f’published message id {future.result()}’) <br> time.sleep(10)</blockquote><p><strong>Create a cloud function that receives the Publisher message.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*zWZKHjO4hnRFoGBnyd60QA.png" /></figure><p><strong>Sample Python Code to receive pub sub message</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_YczB87WCQmy0X3XTua4RA.png" /></figure><p>Function Execution Log</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*AkRW0oTbYfOZJK9f08sMBg.png" /></figure><p>The PubSub messaging service can be used to perform small to medium micro services depends on the requirement.</p><p>This is my first post.All suggestions are invited :-)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45d2fe9ab087" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>