Train Neural Networks without Draining your Pocket: Data Pipeline in TensorFlow and Why it Fails for Heavy Image Datasets

Understand the standard TensorFlow data pipeline how it differs from PyTorch. Observe why they fail to handle massive image datasets.

I’m guessing you are here for one of the two reasons,

• You’ve been trying to train a vision model using TensorFlow and your code crashes while loading the dataset
• You’ve read Part 1 and Part 2 of creating custom heavy-dataset friendly data pipeline for your PyTorch model and you’re curious to see if the same can be done in TensorFlow

Let’s explore if TensorFlow is sturdy enough to handle painfully large datasets.

A QUICK COMPARISON OF PYTORCH AND TENSORFLOW

• Code Style: PyTorch is essentially dynamic which means every operation is executed immediately and the computational graph is built on the go. TensorFlow 2.x (you’ll rarely come across code written in 1.x now) has also enabled eager execution by default but it is still possible to swap to static graphs (which we will explore as one of approaches to speed up training in a later blog) with basic changes in the code. Why do we care about this? Because dynamic execution indicates that errors are easier to debug!