Train Neural Networks without Draining your Pocket: Custom Data Pipeline to Load Large Datasets (PyTorch Version)

Use your knowledge of PyTorch’s backend to create custom data pipelines that can handle massive 3D data volumes with ease. Solution to the previous blog.

If you’re here because you’ve read about Why your Regular Data Pipeline Fails to handle Large Datasets (PyTorch Version) and you’re waiting to see how the cliffhanger we left off at resolves, then welcome back!

If you’re not,

A QUICK RECAP

In the previous part, we had a look at how standard Datasets and Dataloaders work. We tested it out with some image datasets (including a small detour to talk about the Imagefolder class. We also discussed how lazy loading and how PyTorch already includes strategies to make this resource friendly.

It was all going so well till we decided to try things out with the Vesuvius Dataset and this happened,

Yes, this is a memory problem and given which cell causes it indicates that the data is too massive to fit in your RAM.