PinnedInBeyond Engineering.byKeven Pinto·Apr 12, 2022Dataproc Serverless & Airflow 2 Powered Event Driven PipelinesIn my previous post, I demonstrated how one can get a Dataproc Serverless pipeline up and running from the CLI. In this post, we’ll look at…
PinnedInBeyond Engineering.byKeven Pinto·Mar 24, 2022Running pyspark jobs on Google Cloud using Serverless DataprocRun Spark batch workloads without having to bother with the provisioning and management of clusters!.A response icon1A response icon1
InBeyond Engineering.byKeven Pinto·Mar 23, 2023A review of the Google Datastream Service - Cloud SQL (PostgreSQL) to BigQueryDatastream introduced the ability to seamlessly replicate data from Cloud SQL to BigQuery late in 2022. Given that (near) real time…
InBeyond Engineering.byKeven Pinto·Mar 14, 2023Using Datastream as a Data Sync and DR toolIn my last blog we looked at how near real time BigQuery data sync across two regions could not be achieved using the dataset copy service…
InBeyond Engineering.byKeven Pinto·Jan 26, 2023BigQuery Cross-Region Replication Gotchas2022 was the fifth warmest year on record. The July heatwave pushing temperatures to 40°C in parts of the UK was so brutal, that it was a…
InBeyond Engineering.byKeven Pinto·Oct 27, 2022A Centralised Approach to CICD of DAGs on Google Cloud Composer with Google Cloud Build — Part 2In Part 1 of this blog we setup the infrastructure and other building blocks of our CICD pipeline, in this part we will show how we…
InBeyond Engineering.byKeven Pinto·Oct 20, 2022A Centralised Approach to CICD of DAGs on Google Cloud Composer with Google Cloud Build — Part 1In this 2 part Blog, we shall look at how we implemented CICD of DAGs on Google Cloud Composer using Google Cloud Build as our CICD…
InBeyond Engineering.byKeven Pinto·Aug 11, 2022Build, Test and Deploy Dataflow Flex Templates in Google CloudDataflow Flex Templates are a great way to package and distribute your DataFlow Pipeline. That being said, many engineers trying to…A response icon4A response icon4
InBeyond Engineering.byKeven Pinto·Jun 19, 2022Hive to BigQuery: Orchestrating the House Move of the WarehouseHive to BigQuery is a common data migration path and there are plenty of ways to get there. This article describes the approach we took…