<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samuel Knight on Medium]]></title>
        <description><![CDATA[Stories by Samuel Knight on Medium]]></description>
        <link>https://medium.com/@sammyknight?source=rss-e1018c74ab90------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*w3FoUqBaxWK_J_S8wz161g.jpeg</url>
            <title>Stories by Samuel Knight on Medium</title>
            <link>https://medium.com/@sammyknight?source=rss-e1018c74ab90------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:24:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sammyknight/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cutting the Time of Building Neural Models through Distributed Training]]></title>
            <link>https://medium.com/net-ai-insights/cutting-the-time-of-building-neural-models-through-distributed-training-782e5ee6edf9?source=rss-e1018c74ab90------2</link>
            <guid isPermaLink="false">https://medium.com/p/782e5ee6edf9</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Samuel Knight]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 10:54:56 GMT</pubDate>
            <atom:updated>2023-04-12T10:56:41.365Z</atom:updated>
            <content:encoded><![CDATA[<p>One must experience being locked in a trance watching the hypnotically slow updates of a progress bar to truly know the pain of long-running compute jobs. Luckily for most of us in the machine learning space, there is a simple way to reduce this headache: distributed training on multiple GPUs. While incurring a higher capital expenditure, due to the extra hardware required, faster experiment iteration can translate to higher flexibility and productivity for any ML team. At <a href="https://www.netai.tech/">Net AI</a>, we use distributed training workflows to train cutting-edge traffic decomposition and forecasting models, which mobile network operators can use to improve the management of resources in their infrastructure.</p><p>This technical blog focuses on migrating from single GPU training to <strong>synchronous </strong>distributed training on a single machine with multiple GPUs using TensorFlow’s <a href="https://www.tensorflow.org/guide/distributed_training#mirroredstrategy"><strong><em>MirroredStrategy</em></strong></a>. <em>MirroredStrategy</em> replicates model-related variables (parameters, optimizer, loss function, etc.) across all GPUs. TensorFlow also supports <strong>asynchronous</strong> distributed training via <a href="https://www.tensorflow.org/guide/distributed_training#parameterserverstrategy"><em>ParameterServerStrategy</em></a> and multi-worker distributed training via <a href="https://www.tensorflow.org/guide/distributed_training#multiworkermirroredstrategy"><em>MultiWorkerMirroredStrategy</em></a> for other use cases.</p><p>💡 A best-practice training workflow should include a number of additional steps (data exploration, model validation, early-stopping, hyper-parameter tuning, etc.) that are skipped in this tutorial for brevity.</p><h3>Dataset and Model Setup:</h3><p>Following the toy example laid out in the <a href="https://www.tensorflow.org/tutorials/images/classification">TensorFlow documentation</a>, we will be training a CNN to classify five types of flowers. The ResNet architecture was chosen for our CNN, as it is quick to set up and sufficiently complex to demonstrate performance benefits with multiple GPUs. The training server we are using is equipped with four NVIDIA V100 GPUs.</p><p>First, we create an object classification dataset using TensorFlow’s dataset utilities:</p><pre>dataset_url = &quot;https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz&quot;<br>data_dir = tf.keras.utils.get_file(&#39;flower_photos&#39;, origin=dataset_url, untar=True)<br>data_dir = pathlib.Path(data_dir)<br><br>batch_size = 64<br>img_height = 224<br>img_width = 224<br><br>train_ds = tf.keras.utils.image_dataset_from_directory(<br>  data_dir,<br>  seed=123,<br>  image_size=(img_height, img_width),<br>  batch_size=batch_size<br>)</pre><p>Next, we initialise a ResNet50 CNN model for training:</p><pre>num_classes = len(train_ds.class_names)<br><br>def get_model():<br>  resnet = tf.keras.applications.resnet.ResNet50(<br>    weights=None,<br>    include_top=True,<br>    classes=num_classes,<br>    classifier_activation=None<br>  )<br>  return tf.keras.Sequential([<br>    tf.keras.layers.Lambda(tf.keras.applications.resnet50.preprocess_input, input_shape=(224, 224, 3)),<br>    resnet<br>  ])<br><br>def get_compiled_model():<br>  model = get_model()<br>  model.compile(<br>    optimizer=&#39;adam&#39;,<br>    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),<br>    metrics=[&#39;accuracy&#39;]<br>  )<br>  return model<br><br>model = get_compiled_model()</pre><h3>Single GPU Training:</h3><p>TensorFlow automatically detects that GPU compute capability is available and will pick a single GPU to use by default. We call <em>model.fit()</em> to train our model, recording the time taken:</p><pre># Run single epoch of training to avoid cold start affecting time<br>model.fit(train_ds, epochs=1)<br><br>start_time = time.time()<br>model.fit(train_ds, epochs=EPOCHS_TO_TRAIN)<br>print(f&quot;Training for {EPOCHS_TO_TRAIN} epochs took {time.time() - start_time}s&quot;)</pre><p>Training the network for 10 epochs takes just under 2 minutes:</p><pre>Epoch 10/10<br>58/58 [======================] - 11s 189ms/step - loss: 0.6831 - accuracy: 0.7499<br>Training for 10 epochs took 110.58388566970825s</pre><p>We can verify that the training process is only utilizing a single GPU by invoking <em>nvidia-smi</em>:</p><figure><img alt="nvidia-smi output demonstrating that a single GPU was utilised during training." src="https://cdn-images-1.medium.com/max/643/1*_9xm-SDUiF-nrvWDgx4ypg.png" /></figure><h3>Distributed training:</h3><p>To fully utilise the compute resources at our disposal, we will distribute work across the four GPUs on our training machine. The high-level approach for each step of training is detailed as follows:</p><p>Distribute an equal portion of the ‘global’ training batch to each of the GPUs.</p><p>💡 One of the core assumptions of this default distribution approach is that by splitting the global batch into equally sized sub-batches, the time taken for each GPU to process its sub-batch will be approximately equal.</p><figure><img alt="The global batch is split up into sub-batches of equal size and distributed across the GPUs." src="https://cdn-images-1.medium.com/max/1024/1*K4QTsklfUYDCwAuTRZgXiQ.png" /></figure><p>Each GPU executes the training loop with its local sub-batch of data, acquiring local losses and computing local gradients.</p><p>⚠️ As the local gradients will be summed to acquire the global gradient, it is important to ensure that the local losses are scaled appropriately by dividing by the global batch size. <em>model.fit()</em> handles this automatically, but we will have to do it ourselves later when defining a custom training loop.</p><figure><img alt="Each GPU executes a local training loop on its sub-batch, acquiring a local loss value and obtaining gradients from the loss." src="https://cdn-images-1.medium.com/max/746/1*uuuS-5W8_jeZnX5fy5KqeQ.png" /></figure><p>The local gradients are summed to produce a global gradient which is broadcast to all GPUs.</p><figure><img alt="Local gradients are summed and reduced to a single global gradient which is distributed to all GPUs." src="https://cdn-images-1.medium.com/max/1024/1*wNMgKVcG9Cd_QOGYNMpgTg.png" /></figure><p>Finally, all GPUs update their model parameters using the global gradients, resulting in identical model parameters across all GPUs. This approach is defined as <strong>synchronous</strong> precisely because the GPUs must wait for each other to complete all of the above steps before beginning the next training step.</p><h4>Adjusting the batch size:</h4><p>Training on four GPUs using the above approach means that each GPU will receive sub-batches of size <em>batch_size / 4.</em> To keep comparisons fair, we resample our dataset with a global batch size of <em>batch_size × 4</em>.</p><pre>num_gpus = len(tf.config.list_physical_devices(&#39;GPU&#39;))<br>global_batch_size = batch_size * num_gpus<br><br>multi_gpu_train_ds = tf.keras.utils.image_dataset_from_directory(<br>  data_dir,<br>  seed=123,<br>  image_size=(img_height, img_width),<br>  batch_size=global_batch_size<br>)</pre><h4>Using <em>model.fit()</em>:</h4><p>To replicate model parameters and other variables (optimizer, loss, etc.) across all GPUs, the model needs to be created inside the scope of <em>MirroredStrategy</em>. The <em>model.fit()</em> call is strategy aware, meaning it automatically distributes the data across GPUs during training if the model has been defined in the strategy’s scope.</p><pre># Train utilising all GPUs on the system<br>strategy = tf.distribute.MirroredStrategy()<br>with strategy.scope():<br>  multi_gpu_model = get_compiled_model()<br><br># Run single epoch of training to avoid cold start affecting time<br>multi_gpu_model.fit(multi_gpu_train_ds, epochs=1)<br><br>start_time = time.time()<br>multi_gpu_model.fit(multi_gpu_train_ds, epochs=10)<br>print(f&quot;Training for {EPOCHS_TO_TRAIN} epochs took {time.time() - start_time}s&quot;)</pre><p>Now that we are distributing training across four GPUs, we see training time fall to 41.66 seconds, a decrease of approximately <strong>61%</strong>.</p><pre>Epoch 10/10<br>15/15 [======================] - 4s 232ms/step - loss: 0.4177 - accuracy: 0.8466<br>Training for 10 epochs took 41.65961089134216s</pre><p>💡 Many factors can affect the reductions in training time when distributing across GPUs including sub-batch balancing, CPU bottlenecks in data processing, the saturation of GPU memory, overhead in cross-GPU communication, etc.</p><p><em>n</em>vidia-smi allows us to verify that all the GPUs are being used.</p><figure><img alt="nvidia-smi output demonstrating that all four GPUs were utilised during training." src="https://cdn-images-1.medium.com/max/641/1*km_71xr14DRi3EGeh4ehAw.png" /></figure><h4>Using a custom training loop:</h4><p>In some cases, it is useful to have finer-grained control over the training procedure than the <em>model.fit()</em> API provides. In order to take such control, we will need to write a <a href="https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch">custom training loop</a>.</p><p>⚠️ There are three requirements that need to be satisfied to distribute training using <em>MirroredStrategy</em> in a custom training loop:</p><ol><li>Creating variables in the strategy’s scope: Any variable that will need to be used across GPUs should be instantiated within the <em>strategy.scope</em> context. This includes models, optimizers, metrics, and losses.</li><li>Correctly calculating losses: Losses need to be scaled by the global batch size so that the gradients derived from these losses can be sum aggregated across GPUs.</li><li>Manually distributing the dataset: The distribution strategy’s experimental_distribute_dataset function ensures that each batch of data is distributed equally across GPUs during training.</li></ol><p>We start by initialising the variables that we need to be distributed across all GPUs inside of the strategy’s scope. The loss object is defined with no reduction and our loss function scales by the global batch size to ensure losses are correctly calculated.</p><pre>strategy = tf.distribute.MirroredStrategy()<br>with strategy.scope():<br>  model = get_convolutional_model()<br>  optimizer = tf.keras.optimizers.Adam()<br>  train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name=&#39;train_accuracy&#39;)<br><br>  loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)<br>  def loss_fn(labels, predictions):<br>    per_example_loss = loss_object(labels, predictions)<br>    return tf.nn.compute_average_loss(per_example_loss, global_batch_size=global_batch_size)</pre><p>Next, we define our training functions:</p><pre>def train_step(train_x, train_y):<br>    with tf.GradientTape() as tape:<br>      y_pred = model(train_x, training=True)<br>      loss = loss_fn(train_y, y_pred)<br>    grads = tape.gradient(loss, model.trainable_variables)<br>    optimizer.apply_gradients(zip(grads, model.trainable_variables))<br>    train_accuracy.update_state(train_y, y_pred)<br>    return loss<br><br>@tf.function<br>def distributed_training_epoch(dataset):<br>  total_loss = 0.0<br>  num_batches = 0.0<br>  for train_x, train_y in dataset:<br>    per_replica_losses = strategy.run(train_step, args=(train_x, train_y))<br>    total_loss += strategy.reduce(<br>        tf.distribute.ReduceOp.SUM, per_replica_losses, axis=None<br>    )<br>    num_batches += 1<br>  epoch_loss = total_loss / num_batches<br>  print(f&quot;Epoch loss: {epoch_loss}&quot;)<br>  return epoch_loss</pre><p>We manually create a distributed dataset with the default data distribution approach:</p><p>💡 If this approach does not adequately balance sub-batches with respect to per-GPU execution time, use <em>strategy.distribute_datasets_from_function()</em> to manually control how data is batched across GPUs.</p><pre>train_dataset_distributed = strategy.experimental_distribute_dataset(multi_gpu_train_ds)</pre><p>Finally, we construct our training loop:</p><pre># Run single epoch of training to avoid cold start affecting time<br>train_loss = distributed_training_epoch(train_dataset_distributed)<br>train_accuracy.reset_states()<br><br>start_time = time.time()<br>for epoch in range(EPOCHS_TO_TRAIN):<br>  train_loss = distributed_training_epoch(train_dataset_distributed)<br>  train_accuracy.reset_states()<br>print(f&quot;Training for {EPOCHS_TO_TRAIN} epochs took {time.time() - start_time}s&quot;)</pre><p>10 epochs of our custom training loop take 35.88s representing a <strong>67.9%</strong> reduction in training time compared to using a single GPU.</p><pre>Epoch 10/10 -- loss: 0.5216162800788879 - accuracy: 0.8000000011920929<br>Training for 10 epochs took 35.87821805477142s</pre><h3>Closing Remarks:</h3><p>This tutorial has demonstrated that distributed training with TensorFlow can be straightforward to implement and lead to significant reductions in training time. At <a href="https://www.netai.tech/">Net AI</a>, we utilise distributed training to decrease experimentation time, enabling us to innovate faster, as we continue to create cutting-edge solutions for mobile network operators.</p><h4>Source Code:</h4><p>The code for the above tutorial can be found in <a href="https://colab.research.google.com/drive/1B60ewlJ91JUnZFczACMDTZQF_57MlBU_?usp=sharing">this Google Colab notebook</a>.</p><h4>Useful TensorFlow Documentation:</h4><p>Distributed training: <a href="https://www.tensorflow.org/guide/distributed_training">https://www.tensorflow.org/guide/distributed_training</a></p><p>Custom training loop: <a href="https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch">https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=782e5ee6edf9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/net-ai-insights/cutting-the-time-of-building-neural-models-through-distributed-training-782e5ee6edf9">Cutting the Time of Building Neural Models through Distributed Training</a> was originally published in <a href="https://medium.com/net-ai-insights">Net AI Insights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>