How we prepared the test

In order to compare the performance of CPUs vs GPUs vs TPUs for accomplishing common data science tasks, we used the tf_flowers dataset to train a convolutional neural network, and then the exact same code was run three times using the three different backends (CPUs vs GPUs vs TPUs; GPUs were NVIDIA P100 with Intel Xeon 2GHz (2 core) CPU and 13GB RAM. TPUs were TPUv3 (8 core) with Intel Xeon 2GHz (4 core) CPU and 16GB RAM). The accompanying tutorial notebook demonstrates a few best practices for getting the best performance out of your TPU.

For example:

1. Using a dataset of sharded files (e.g., .TFRecord)
2. Using the tf.data API to pass the training data to the TPU
3. Using large batch sizes (e.g. batch_size=128)

By adding these precursory steps to your workflow, it is possible to avoid a common I/O bottleneck that otherwise prevents the TPU from operating at its full potential. You can find additional tips for optimizing your code to run on TPUs by visiting the official Kaggle TPU documentation.

How the hardware performed

The most notable difference between the three hardware types that we tested was the speed that it took to train a model using tf.keras. The tf.keras library is one of the most popular machine learning frameworks because tf.keras makes it easy to quickly experiment with new ideas. If you spend less time writing code then you have more time to perform your calculations, and if you spend less time waiting for your code to run, then you have more time to evaluate new ideas (Figure 2). tf.keras and TPUs are a powerful combination when participating in machine learning competitions!

For our first experiment, we used the same code (a modified version*** of the official tutorial notebook) for all three hardware types, which required using a very small batch size of 16 in order to avoid out-of-memory errors from the CPU and GPU. Under these conditions, we observed that TPUs were responsible for a ~100x speedup as compared to CPUs and a ~3.5x speedup as compared to GPUs when training an Xception model (Figure 3). Because TPUs operate more efficiently with large batch sizes, we also tried increasing the batch size to 128 and this resulted in an additional ~2x speedup for TPUs and out-of-memory errors for GPUs and CPUs. Under these conditions, the TPU was able to train an Xception model more than 7x as fast as the GPU from the previous experiment****.

The observed speedups for model training varied according to the type of model, with Xception and Vgg16 performing better than ResNet50 (Figure 4). Model training was the only type of task where we observed the TPU to outperform the GPU by such a large margin. For example, we observed that in our hands the TPUs were ~3x faster than CPUs and ~3x slower than GPUs for performing a small number of predictions (TPUs perform exceptionally when making predictions in some situations such as when making predictions on very large batches, which were not present in this experiment).

To supplement these results, we note that Wang et. al have developed a rigorous benchmark called ParaDnn [1] that can be used to compare the performance of different hardware types for training machine learning models. By using this method Wang et. al were able to conclude that the performance benefit for parameterized models ranged from 1x to 10x, and the performance benefit for real models ranged from 3x to 6.8x when a TPU was used instead of a GPU (Figure 5). TPUs perform best when combined with sharded datasets, large batch sizes, and large models.

Price considerations when training models

While our comparisons treated the hardware equally, there is a sizeable difference in pricing. TPUs are ~5x as expensive as GPUs ($1.46/hr for a Nvidia Tesla P100 GPU vs $8.00/hr for a Google TPU v3 vs $4.50/hr for the TPUv2 with “on-demand” access on GCP ). If you are trying to optimize for cost then it makes sense to use a TPU if it will train your model at least 5 times as fast as if you trained the same model using a GPU.

We consistently observed model training speedups on the order of ~5x when the data was stored in a sharded format in a GCS bucket then passed to the TPU in large batch sizes, and therefore we recommend TPUs to cost-conscious consumers that are familiar with the tf.data API.

Some machine learning practitioners prioritize the reduction of model training time as opposed to prioritizing the reduction of model training costs. For someone that just wants to train their model as fast as possible, the TPU is the best choice. If you spend less time training your model, then you have more time to iterate upon new ideas. But don’t take our word for it — you can evaluate the performance benefits of CPUs, GPUs, and TPUs by running your own code in a Kaggle Notebook, free-of-charge. Kaggle users are already having a lot of fun and success experimenting with TPUs and text data: check out this forum post that describes how TPUs were used to train a BERT transformer model to win $8,000 (2nd prize) in a recent Kaggle competition.

Which hardware option should you choose?

In summary, we recommend CPUs for their versatility and for their large memory capacity. GPUs are a great alternative to CPUs when you want to speed up a variety of data science workflows, and TPUs are best when you specifically want to train a machine learning model as fast as you possibly can.

You can get better results by optimizing your code for the specific hardware that you are using and we think it would be especially interesting to compare runtimes for code that has been optimized for a GPU to runtimes for code that has been optimized for a TPU. For example, it would be interesting to record the time that it takes to train a gradient-boosted model using a GPU-accelerated library such as RAPIDS.ai and then to compare that to the time that it takes to train a deep learning model using a TPU-accelerated library such as tf.keras.

What is the least amount of time that one can train an accurate machine learning model? How many different ideas can you evaluate in a single day? When used in combination with tf.keras, TPUs allow machine learning practitioners to spend less time writing code and less time waiting for their code to run — leaving more time to evaluate new ideas and improve one’s performance in Kaggle Competitions.

Works Cited

[1] Wang Y, Wei G, Brooks D. Benchmarking TPU, GPU, and CPU Platforms for Deep Learning. 2019. arXiv: 1907.10701.

[2] Kumar S, Bittorf V, et. al. Scale MLPerf-0.6 models on Google TPU-v3 Pods. 2019. arXiv: 1909.09756.

[3] Jouppi N, Young, C, et. al. In-datacenter performance analysis of a tensor processing unit. 2017. 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA).

Footnotes

* CPU types vary according to variability. In addition to the Intel Xeon CPUs, you can also get assigned to either Intel Skylake, Intel Broadwell, or Intel Haswell CPUs. GPUs were NVIDIA P100 with Intel Xeon 2GHz (2 core) CPU and 13GB RAM. TPUs were TPUv3 (8 core) with Intel Xeon 2GHz (4 core) CPU and 16GB RAM).

** Image for Figure 1 from https://cloud.google.com/blog/products/ai-machine-learning/cloud-tpu-breaks-scalability-records-for-ai-inference, with permission.

*** The tutorial notebook was modified to keep the parameters (e.g. batch_size, learning_rate, etc) consistent between the three different backends.

**** CPU and GPU experiments used a batch size of 16 because it allowed the Kaggle notebooks to run from top to bottom without memory errors or 9-hr timeout errors. Only TPU-enabled notebooks were able to run successfully when the batch size was increased to 128.

When to use CPUs vs GPUs vs TPUs in a Kaggle Competition? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction:

Can “citizen scientists” do real science without costly laboratory equipment?

In short: yes. In this blog post we point out an alternative and low-cost approach to solving biological questions that is suited to the aspiring amateur scientist. Specifically, we highlight the benefits of combining low-cost imaging equipment (FoldScope microscopes) and public image data (Kaggle Datasets) with free tools for performing computational data analysis (Kaggle Kernels). Furthermore, we provide a general framework for using these tools to tackle image classification problems in the biological sciences. We hope that these guidelines encourage more data scientists to publish cell biology datasets and to share what they learn.

Part 1: Obtaining image data

High-throughput and super-resolution microscopy methods have been invaluable to the study of cellular biology [1,2], but these technologies are prohibitively expensive for many or most research labs. The good news, however, is that innovative technologies like FoldScope microscopes can dramatically decrease research costs [3]. FoldScope microscopes are literally paper microscopes, made of a paper frame attached to a high-magnification objective lens (Figure 1). These multi-use microscopes have proven capable of imaging infectious agents such as Plasmodium falciparum and Schistosoma haematobium [3,4], and can be attached to the camera of a cellular phone. FoldScope microscopes currently retail for as little as $1.50 each, and their low cost has inspired an active community of citizen scientists.

In this work we acquired new cell images using a FoldScope microscope. We opted to start with commercially prepared slides, sold by the Carolina Biological Supply Company, containing dividing Ascaris lumbricoides cells and hematoxylin staining. Images were acquired using a FoldScope microscope equipped with a 500x magnification lens and a 8-megapixel digital camera (iPhone 5). In this way, we generated a dataset of images of cells in different stages of cellular division with a very minimal investment.

Part 2: Sharing image data

A FoldScope microscope was used to acquire 90 images of dividing cells in hematoxylin stained Ascaris lumbricoides uterus (Figure 2). These images were shared as a public dataset on Kaggle along with some starter code demonstrating how to work with the data. A dataset of only 90 images is inevitably limited and does not present a suitable challenge for the computational environment. As such, we identified a relatively large dataset of labeled images of Jurkat cells in various different stages of cellular division [5] and added it to our Kaggle Dataset as well. With this supplemental dataset of approximately 32,000 cell images, we could sufficiently test the ability of the free Kaggle Kernels product for performing complex computational analyses.

Part 3: Analyzing image data

Deep learning algorithms are exciting in part because of their potential to automate biomedical research tasks [6,7]. For example, deep learning algorithms can be used to automate the time-consuming process of manually counting mitotic structures in breast histopathology images [8,9]. Differences in the rates of cellular division and differences in the amount of time spent in each stage of cellular division are both important differentiators between healthy cells and cancer cells [10,11]. Likewise, cancer cells often form erroneous mitotic structures during cellular division and these erroneous structures can contribute to further disease progression [12,13]. As such, the study of cellular division and and its machinery has led to the development of numerous anti-cancer drugs [14,15]. In this work we use the cell division dataset from Figure 2 to train a deep learning model that can be used to identify cell cycle stages in images of dividing cells (Figure 3).

Here we present a simple approach that combines the Kaggle Kernel with the cell division dataset from Figure 2 in order to train a deep neural network to identify cell cycle stages. The free-of-charge Kaggle Kernel combines the data, the code, and the cloud-based computational environment in such a way that makes the work easy to reproduce. In fact, you can duplicate this work and you can run it in an identical cloud-based computational environment just by pressing the “Fork Kernel” button on the Kaggle website. (Additional discussion on research reproducibility and a set of recommended guidelines can be found here). By using a dataset of 32,266 images to train a deep neural network, we hope we’ve shown how well the free Kaggle Kernels environment can perform complex computational analyses that are relevant to the biomedical sciences.

The ML model that was trained under this new approach gave results that were comparable to the original analysis (Figure 4). Interestingly, both models still have room for improvement in identifying some of the more rarely-observed cell cycle stages, but this could likely be corrected for by generating additional data. The code in Figure 4B is meant to be generalizable and should work well with diverse image types and image categories: the fastai.ImageDataBunch.from_folder() function can be used to load and process any compatible image, and the fastai.create_cnn() function can be used to automatically learn new model features.

# Code to generate Figure 4A: https://github.com/theislab/deepflow
# Code to generate Figure 4B: (below)

from fastai import *
from fastai.vision import *
from fastai.callbacks.hooks import *
import numpy as np; import pandas as pd
import matplotlib; import matplotlib.pyplot as plt

img_dir='../input/'; path=Path(img_dir)
data=ImageDataBunch.from_folder(path, train=".",valid_pct=0.3,                               ds_tfms=get_transforms(do_flip=True,flip_vert=True,max_rotate=90,max_lighting=0.3),size=224,bs=64,num_workers=0).normalize(imagenet_stats)

learn=create_cnn(data, models.resnet34, metrics=accuracy, model_dir="/tmp/model/")
learn.fit_one_cycle(10)
interp=ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix(figsize=(10,10), dpi=60)

This work describes a reusable framework that can be applied to cell biology datasets to solve image classification problems. The specific image classification problem discussed was the automatic identification of cell cycle stages during cellular division, and the approach was notable because of the low cost of the equipment and because of how easy it was to train the model (with free cloud computing and open-sourced ML algorithms). Data scientists and cell biologists who want to train their own image classification models can easily replicate this approach by creating a dataset of their own FoldScope images, organizing the training data into folders that correspond with each image label, and then attaching that same dataset to a kernel that contains the Fastai code described in Figure 4. We hope that future researchers take advantage of the dataset and the starter code that we have shared in order to quickly fork, modify, and improve upon our models.

Conclusion:

Since the first compound microscopes were built in the 1600s, medical imaging and laboratory equipment has been expensive to produce and to own. Analytical software is much newer but in many cases can be equally expensive. More recently, low-cost tools like the $1.50 FoldScope microscope and the $0.00 Kaggle Kernel have been developed that can perform many of these same functions at a fraction of the cost. This work describes a low-cost and reusable framework that can be applied to cell biology datasets to solve image classification problems. We hope that these guidelines will encourage more data scientists to explore and publish cell biology datasets and to share their results and findings.

Works Cited:

[1] Miller MA, Weissleder R. Imaging of anticancer drug action in single cells. Nature Reviews Cancer. 2017. Vol 17(7): p399–414.

[2] Liu TL, Upadhyayula S, Milkie, et al. Observing the cell in its native state: Imaging subcellular dynamics in multicellular organisms. Science. 2018 Vol 360(6386).

[3] Cybulski J, Clements J, Prakash M. Foldscope: Origami-Based Paper Microscope. PLoS One. 2014; Vol 9(6).

[4] Ephraim R, Duah E, et al. Diagnosis of Schistosoma haematobium Infection with a Mobile Phone-Mounted Foldscope and a Reversed-Lens CellScope in Ghana. Am J Trop Med Hyg. 2015. Vol 92(6): p. 1253–1256.

[5] Eulenberg P, Köhler N, et al. Reconstructing cell cycle and disease progression using deep learning. Nature Communications. 2017. Vol 8(1): p463.

[6] Esteva A, Robicquet A, et al. A guide to deep learning in healthcare. Nature Medicine. 2019. Vol 25: p.24–29.

[7] Topol, Eric J. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine. 2019. Vol 1: p44–56.

[8] Li C, Wang X, Liu W, Latecki LJ. DeepMitosis: Mitosis detection via deep detection, verification and segmentation networks. Medical Image Analysis. 2018. Vol 45:

[9] Saha M, Chakraborty C, Racoceanu D. Efficient deep learning model for mitosis detection using breast histopathology images. Computational Medical Imaging and Graphics. 2018. Vol 64: p29–40.

[10] Sherr CJ. Cancer cell cycles. Science. 1996. Vol 274(5293): p1672–7.

[11] Visconti R, Monica RD, Grieco D. Cell cycle checkpoint in cancer: a therapeutically targetable double-edged sword. J Exp Clin Cancer Res. 2016. Vol 35: p153.

[12] Milunović-Jevtić A, Mooney P, et al. Centrosomal clustering contributes to chromosomal instability and cancer. Current Opinion in Biotechnology. 2016 Vol 40: p113–118.

[13] Bakhoum SF, Cantley LC. The Multifaceted Role of Chromosomal Instability in Cancer and Its Microenvironment. Cell. 2018 Vol 174(6): p1347–1360.

[14] Florian S, Mitchison TJ. Anti-Microtubule Drugs. Methods in Molecular Biology. 2016 Vol 1413: p403–411.

[15] Steinmetz MO, Prota AE. Microtubule-Targeting Agents: Strategies To Hijack the Cytoskeleton. Trends in Cell Biology. 2018 Vol 28(10): p776–792.

[16] The source for the images in Figure 3A, Figure 3B, and the Cover Photo can be found in each respective hyperlink.

Low-Cost Cell Biology Experiments for Data Scientists was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.