<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amandeep Saluja on Medium]]></title>
        <description><![CDATA[Stories by Amandeep Saluja on Medium]]></description>
        <link>https://medium.com/@saluja.amandeep?source=rss-743593c1f100------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Q7mM5Fdyz6GgqCr9-AibAA.jpeg</url>
            <title>Stories by Amandeep Saluja on Medium</title>
            <link>https://medium.com/@saluja.amandeep?source=rss-743593c1f100------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:11:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saluja.amandeep/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GCS to BigQuery via Workflows]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-workflows-48414dea1525?source=rss-743593c1f100------2"><img src="https://cdn-images-1.medium.com/max/600/1*8wsrM8ns0QMHjPvw9CUV0g.png" width="600"></a></p><p class="medium-feed-snippet">Czesc &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-workflows-48414dea1525?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-workflows-48414dea1525?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/48414dea1525</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[gcp-workflows]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 01:40:53 GMT</pubDate>
            <atom:updated>2023-11-27T01:40:53.845Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery via Cloud Composer: Part 4 (Create Cloud Function to Trigger Airflow DAG)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-4-create-cloud-function-to-trigger-airflow-dag-ca840e342961?source=rss-743593c1f100------2"><img src="https://cdn-images-1.medium.com/max/601/1*guTIR71YIHZvBVX8iFRYyw.png" width="601"></a></p><p class="medium-feed-snippet">Yahsoo &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-4-create-cloud-function-to-trigger-airflow-dag-ca840e342961?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-4-create-cloud-function-to-trigger-airflow-dag-ca840e342961?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca840e342961</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cloud-composer]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Sun, 26 Nov 2023 01:29:00 GMT</pubDate>
            <atom:updated>2023-11-26T01:38:18.663Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery via Cloud Composer: Part 3 (Create and Deploy Airflow DAG)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-3-create-and-deploy-airflow-dag-b489fd384d49?source=rss-743593c1f100------2"><img src="https://cdn-images-1.medium.com/max/601/1*guTIR71YIHZvBVX8iFRYyw.png" width="601"></a></p><p class="medium-feed-snippet">Shalom &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-3-create-and-deploy-airflow-dag-b489fd384d49?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-3-create-and-deploy-airflow-dag-b489fd384d49?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/b489fd384d49</guid>
            <category><![CDATA[cloud-functions]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cloud-composer]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Sat, 25 Nov 2023 14:49:19 GMT</pubDate>
            <atom:updated>2023-11-26T01:37:40.773Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery via Cloud Composer: Part 2 (Setup Cloud Composer Environment)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Zdravstvuyte &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-2-setup-cloud-composer-environment-5871333cc7f1?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-2-setup-cloud-composer-environment-5871333cc7f1?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/5871333cc7f1</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[cloud-composer]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Fri, 24 Nov 2023 15:12:39 GMT</pubDate>
            <atom:updated>2023-11-26T01:37:14.748Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery via Cloud Composer: Part 1 (Overview)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-1-overview-2ba74666f0bd?source=rss-743593c1f100------2"><img src="https://cdn-images-1.medium.com/max/640/1*ElfUs191cm6ZRpNVGaD8Ig.png" width="640"></a></p><p class="medium-feed-snippet">Konnichiwa &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-1-overview-2ba74666f0bd?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-1-overview-2ba74666f0bd?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ba74666f0bd</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cloud-functions]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-composer]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Thu, 23 Nov 2023 22:40:30 GMT</pubDate>
            <atom:updated>2023-11-26T01:36:54.470Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery via Cloud Function]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">&#x928;&#x92E;&#x938;&#x94D;&#x924;&#x947; &#x1F64F;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-function-52ed9019c85d?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-function-52ed9019c85d?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/52ed9019c85d</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cloud-functions]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Thu, 09 Nov 2023 15:55:45 GMT</pubDate>
            <atom:updated>2023-11-09T15:55:45.857Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery via Dataproc Serverless: Part 2 (Development)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Ciao &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-dataproc-serverless-part-2-development-c3b959858801?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-dataproc-serverless-part-2-development-c3b959858801?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3b959858801</guid>
            <category><![CDATA[dataproc]]></category>
            <category><![CDATA[dataproc-serverless]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Wed, 08 Nov 2023 21:20:05 GMT</pubDate>
            <atom:updated>2023-11-08T21:20:05.562Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery via Dataproc Serverless: Part 1 (Overview)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-dataproc-serverless-part-1-overview-221bbc833021?source=rss-743593c1f100------2"><img src="https://cdn-images-1.medium.com/max/705/1*IY7mVB5lu6S4zs8fSjjwBA.png" width="705"></a></p><p class="medium-feed-snippet">Bonjour &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-dataproc-serverless-part-1-overview-221bbc833021?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-dataproc-serverless-part-1-overview-221bbc833021?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/221bbc833021</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[dataproc]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Wed, 08 Nov 2023 17:23:21 GMT</pubDate>
            <atom:updated>2023-11-08T21:21:02.877Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[XLSX to CSV GCP Cloud Function]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Hola &#x1F44B;</p><p class="medium-feed-link"><a href="https://medium.com/@saluja.amandeep/xlsx-to-csv-gcp-cloud-function-50839d88b7a5?source=rss-743593c1f100------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@saluja.amandeep/xlsx-to-csv-gcp-cloud-function-50839d88b7a5?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/50839d88b7a5</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cloud-functions]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Tue, 31 Oct 2023 17:36:04 GMT</pubDate>
            <atom:updated>2023-10-31T17:36:04.452Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[GCS to BigQuery Pipeline via Different GCP Services]]></title>
            <link>https://medium.com/@saluja.amandeep/gcs-to-bigquery-pipeline-via-different-gcp-services-a48a342e425e?source=rss-743593c1f100------2</link>
            <guid isPermaLink="false">https://medium.com/p/a48a342e425e</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Amandeep Saluja]]></dc:creator>
            <pubDate>Tue, 31 Oct 2023 15:49:51 GMT</pubDate>
            <atom:updated>2023-11-27T01:41:24.243Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*DEdhzGdJyTOZ7dsXsgeeCg.png" /></figure><p>Welcome back to another big Data Engineering project we are going to be building together.</p><p>In this post, I’m going to be listing (and linking) out all Medium articles I’m going to publish in coming days utilizing different GCP Services. The Goal is to get familiar with different services and see how it works.</p><blockquote>The goal of all upcoming projects will be the same. Process the Excel file dropped into the bucket and load data into BigQuery.</blockquote><p>Why Excel? Well, every business uses it on the daily basis. And we already have lot of pre-built templates available to process csv or parquet files. There are lot of challenges we are going to come across when building data pipelines for Excel. So, buckle up :)</p><h3>Technologies Used</h3><ol><li><a href="https://airflow.apache.org/">Apache Airflow</a></li><li><a href="https://beam.apache.org/">Apache Beam</a></li><li><a href="https://spark.apache.org/">Apache Spark</a></li><li><a href="https://www.docker.com/">Docker</a></li><li>GCP Services<br>- <a href="https://cloud.google.com/bigquery/docs/introduction">BigQuery</a><br>- <a href="https://cloud.google.com/composer/docs/composer-2/composer-overview">Cloud Composer (Apache Airflow)</a><br>- <a href="https://cloud.google.com/functions/docs/concepts/overview">Cloud Functions</a><br>- <a href="https://cloud.google.com/storage/docs/introduction">Cloud Storage</a><br>- <a href="https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates">Dataflow Flex Template (Apache Beam)</a><br>- <a href="https://cloud.google.com/dataflow/docs/guides/templates/provided/cloud-storage-to-bigquery">Dataflow Google Provided Template (Apache Beam)</a><br>- <a href="https://cloud.google.com/dataproc-serverless/docs/overview">Dataproc Serverless (Apache Spark)</a> <br>- <a href="https://cloud.google.com/eventarc/docs/overview">Eventarc</a><br>- <a href="https://cloud.google.com/workflows/docs/overview">Workflows</a><br>- <a href="https://cloud.google.com/iam/docs/workload-identity-federation">Workload Identity Federation</a></li><li><a href="https://docs.github.com/en/actions">GitHub Actions</a></li><li><a href="https://www.python.org/">Python</a></li><li><a href="https://www.terraform.io/">Terraform</a></li></ol><p>Below are links to all the resources I have (and will be) published.</p><p>Before we start anything, <strong>we will be working on creating a generic </strong><a href="https://medium.com/@saluja.amandeep/xlsx-to-csv-gcp-cloud-function-50839d88b7a5"><strong>HTTP Cloud Function (XLSX to CSV Cloud Function)</strong></a> so that it can be used across all pipelines. The purpose to create this is Dataproc Serverless and Dataflow does not have pre-built templates that handles Excel as Input.</p><h3>GCS to BigQuery via Dataproc Serverless</h3><ol><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-dataproc-serverless-part-1-overview-221bbc833021">Overview</a></li><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-dataproc-serverless-part-2-development-c3b959858801">Dataproc Job Submit Cloud Function (Cloud Event Trigger)</a></li></ol><h3>GCS to BigQuery via Cloud Functions</h3><ol><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-function-52ed9019c85d">Overview, Setup, and Deployment</a></li></ol><h3>GCS to BigQuery via Cloud Composer</h3><ol><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-1-overview-2ba74666f0bd">Overview</a></li><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-2-setup-cloud-composer-environment-5871333cc7f1">Setup Cloud Composer Environment</a></li><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-3-create-and-deploy-airflow-dag-b489fd384d49">Create and Deploy Airflow DAG</a></li><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-cloud-composer-part-4-create-cloud-function-to-trigger-airflow-dag-ca840e342961">Create Cloud Function to Trigger Airflow DAG</a></li></ol><h3>GCS to BigQuery via Workflows</h3><ol><li><a href="https://medium.com/@saluja.amandeep/gcs-to-bigquery-via-workflows-48414dea1525">Overview, Setup, and Deployment</a></li></ol><h3>GCS to BigQuery via Dataflow Google Provided Template</h3><ol><li>Overview</li><li>Dataflow Job Submit Cloud Function (Cloud Event Trigger)<br>a. Function Setup<br>b. Terraform Setup<br>c. Deployment via GitHub Actions</li><li>Troubleshooting</li></ol><h3>GCS to BigQuery via Dataflow Flex Template</h3><ol><li>Overview</li><li>Flex Template<br>a. Pipeline Setup<br>b. Terraform Setup<br>c. Docker Setup<br>d. Deployment via GitHub Actions</li><li>Dataflow Job Submit Cloud Function (Cloud Event Trigger)<br>a. Function Setup<br>b. Terraform Setup<br>c. Deployment via GitHub Actions</li><li>Troubleshooting</li></ol><p>There are more services I’m going to explore. I will keep them updated here for your reference. If you have some other GCP services I can utilize, please let me know in the comments below.</p><p>Okay. Lets start this journey ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a48a342e425e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>