Google Cloud Dataflow
Trimming down the cost of running Google Cloud Dataflow Classic
1. Reducing the Disk size: By default, the disk size for the dataflow pipeline is set to 250GB for a batch pipeline and 400GB for a streaming pipeline. You can specify disk size as — diskSizeGb=30
2. Specifying a custom machine type: — workerMachineType=custom-8–7424 The value would correspond to 8 cores and 7424 MB of memory and you can tweak this according to your need.
3. Disabling public IPs:— usePublicIps=false
4. Keeping your GCP services in the same region:To set the region while deploying your Dataflow pipeline, you can add the following execution parameter: — region=us-central
5. Horizontal scaling: If you are trying to figure out whether you want to enable auto-scaling or not or want to start without autoscaling: — autoscalingAlgorithm=NONE — numWorkers=3
Dataflow Prime:
Dataflow Prime helps you to customize your resources as needed for each stage of the pipeline, unlike dataflow classic.
To use Dataflow Prime, you can reuse your existing pipeline code and also enable the Dataflow Prime option either through Cloud Shell or programmatically.
Dataflow Prime features:
Get Arsh’s stories in your inbox
Join Medium for free to get updates from this writer.
The following is the list of supported Dataflow Prime features for different kinds of pipelines:
- Vertical Auto Scaling (memory). Supports streaming pipelines in Python and Java.
- Right Fitting (Dataflow Prime resource hints). Supports batch pipelines in Python and Java.
- Job Visualizer. Supports batch pipelines in Python and Java.
- Smart Recommendations. Supports both streaming and batch pipelines in Python and Java.
- Data Pipelines. Supports both streaming and batch pipelines in Python and Java.
Below is the comparison between Dataflow Classic and Dataflow Prime
Use the following links to get more information on Dataflow which will help you Benchmark Dataflow jobs for cost, performance, and Capacity planning.
https://cloud.google.com/dataflow/docs/guides/enable-dataflow-prime#enable-prime









