CuriousTechie·Sep 30POC 1 — GCP Data Platform: BigQuery, Dataproc & DataplexTags: gcp bigquery dataproc dataplex pubsub
InGoogle Cloud - CommunitybyShashank Tripathi·Aug 19From Prompt Engineering to Agentic Workflow: Building Intelligent Systems on Dataproc Serverless…In the world of Big Data, the journey from writing code to running a production Spark job has traditionally been a manual, multi-step…
Allan McAleavy·Jul 1Monitoring Queues in Dataproc & Apache Spark — yarn_top_lightWhen looking at YARN scheduling behaviour, yarn top gives a useful cluster-wide view of allocated memory, allocated vcores, running…
Allan McAleavy·Jul 1Deterministic vs Burstable YARN Queues on DataprocYARN Capacity Scheduler looks simple on paper: define queue percentages, submit workloads, and expect each queue to receive its share.
Allan McAleavy·Jun 17Can We Predict How Many Spark Tasks an Iceberg Ingestion Will Create?I have recently started looking at how Apache Spark(Dataproc) works with Iceberg as a data lake, with a view to better understanding it…
InGoogle Cloud - CommunitybyMurli Krishnan·May 29The BigQuery Iceberg Handbook: Data Integration Patterns — Part 4 Apache Iceberg REST CatalogGCP platform introduced support for REST catalog for Biglake Metastore in late 2025. The Biglake Metastore has now been renamed as…
InGoogle Cloud - CommunitybyMurli Krishnan·May 29The BigQuery Iceberg Handbook: Data Integration Patterns — Part 3 Apache Iceberg Catalog for…In this blog, we will be discussing on Lakehouse Runtime Catalog — Apache Iceberg Catalog for Bigquery.
Alberto López Serna·May 10Part 1: F1 Telemetry and Tuning for your Spark cluster: the BigQuery Log Analytics setup that costs…I can do auto tuning (every day) for +10000 spark jobs and ~400 clusters. Seamlessly SRE, cost tracking/optimisations and serious balance…
RAJAN GUPTA·Mar 1Dataproc Vs Dataflow: GCP ServicesProcessing data is something data engineers do day in and day out. When building pipelines on Google Cloud, two major services often come…
