InInside LeaguebyAlex Fragotsis·Dec 26, 2021Streaming Data to BigQuery with Dataflow and Updating the Schema in Real-TimeIn our previous story, we saw how to stream data to Big Query and also add new columns when needed. This solution though is not really…A response icon4A response icon4
Alex Fragotsis·Jun 4, 2021Authenticated calls to cloud functions with PythonThe past few weeks I developed and deployed a cloud function that is supposed to get called only by authorized users/service accounts and…A response icon1A response icon1
InInside LeaguebyAlex Fragotsis·Jan 17, 2021Loading complex JSON files in RealTime to BigQuery from PubSub using Dataflow and updating the…In my previous post, I explained how to stream data from Salesforce to PubSub in real-time. The next logical step would be to store the…A response icon6A response icon6
InInside LeaguebyAlex Fragotsis·Dec 12, 2020Real-Time Streaming Salesforce Updates to PubSubA response icon2A response icon2
InThe StartupbyAlex Fragotsis·Aug 29, 2020Pyspark: How to Modify a Nested Struct FieldIn our adventures trying to build a data lake, we are using dynamically generated spark cluster to ingest some data from MongoDB, our…A response icon3A response icon3
Alex Fragotsis·Jul 4, 2020Schedule Dataflow Templates with AirflowOk, so, we’ve written our Dataflow Template with Python, now what? We want to schedule it to run daily and we’re going to use Airflow for…
InAnalytics VidhyabyAlex Fragotsis·May 31, 2020Transform JSON to CSV from Google bucket using a Dataflow Python pipelineIn this article, we will try to transform a JSON file into a CSV file using dataflow and pythonA response icon3A response icon3