<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neil Tenenholtz on Medium]]></title>
        <description><![CDATA[Stories by Neil Tenenholtz on Medium]]></description>
        <link>https://medium.com/@ntenenz?source=rss-c8b4070d70a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OR9-0fCd398nw1Y96hTBjg.jpeg</url>
            <title>Stories by Neil Tenenholtz on Medium</title>
            <link>https://medium.com/@ntenenz?source=rss-c8b4070d70a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:36:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ntenenz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Distributed Tensorflow]]></title>
            <link>https://medium.com/@ntenenz/distributed-tensorflow-2bf94f0205c3?source=rss-c8b4070d70a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bf94f0205c3</guid>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Neil Tenenholtz]]></dc:creator>
            <pubDate>Sun, 01 Apr 2018 13:07:46 GMT</pubDate>
            <atom:updated>2018-04-01T13:25:20.834Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Note: This was previously posted on </em><a href="http://www.ccds.io/"><em>www.ccds.io</em></a><em>.</em></p><p><a href="https://www.tensorflow.org/">Tensorflow</a> is a popular open-source library designed for numerical computation, most commonly the training and serving of neural networks. In this framework, computation is described via data flow graphs, which offer a large degree of flexibility in the structure and placement of operations. This allows for parallelizing computation across multiple workers, which is often beneficial in the training of neural networks given the large number of training data that must be processed. Furthermore, such parallelization may sometimes be required if the size of the model becomes sufficiently large. In this post, we will explore the mechanisms through which computation in TensorFlow can be distributed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uvMPEwhmI7JKbgoBO6SJOQ.png" /><figcaption>A sample Tensorflow graph</figcaption></figure><p>When splitting the training of a neural network across multiple compute nodes, two strategies are commonly employed: data parallelism and model parallelism. In the former, individual instances of the model are created on each node and fed different training samples; this allows for higher training throughput. Conversely, in model parallelism, a single instance of the model is split across multiple nodes allowing for larger models, ones which may not necessarily fit in the memory of a single node, to be trained. If desired, these two strategies can also be composed resulting in multiple instances of a given model with each instance spanning multiple nodes. In this post, we will focus on data parallelism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*q_srpj5rphbXaphyBr4tUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*yEG-GbigZYKVtYW4SfJ3yQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*aNcIekBGWBd6L7h4Ckgtcg.png" /><figcaption>Examples of different forms of data and model parallelism: data parallelism (left), model parallelism (center), data and model parallelism (right).</figcaption></figure><p>Data parallelism in Tensorflow largely manifests itself in two forms: in-graph replication and between-graph replication. The most significant difference between the two strategies lies in the structure of the compute graph and the consequences that result.</p><p>In-graph replication is commonly thought to be the simpler and more straightforward (but less scalable) of the two approaches. When employing this strategy, a single graph is created on the distributed master that includes all of the replicas residing on its worker devices. As one can imagine, such a graph can grow to be quite large as the number of workers grows, which can adversely impact performance. However, for small systems (e.g., a dual-GPU desktop computer), in-graph replication may be preferred due to its simplicity and transparency.</p><p>Below are snippets demonstrating the fairly painless manner in which in-graph replication can be applied relative to baseline TensorFlow code using a single GPU. Given the issues this approach has with scaling, we will only consider the case of a single-machine, multi-GPU configuration. The differences between the two blocks of code are fairly minor, namely chunking of the input data so that it’s evenly distributed across workers, iterating over the workers where the graph elements are placed on each device, and the concatenation of the results from the various workers. With this small code change, one is able to leverage multiple devices, making this method popular for simpler configurations where scalability is not a significant concern.</p><p>Compare this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df1c82f040eb5848b54b8f44be7a3f48/href">https://medium.com/media/df1c82f040eb5848b54b8f44be7a3f48/href</a></iframe><p>with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecdc1dd830dfdd49d4dd1bf0960f0674/href">https://medium.com/media/ecdc1dd830dfdd49d4dd1bf0960f0674/href</a></iframe><p>These changes can also be visualized by examining the TensorFlow graphs, which are included below. The repeated GPU blocks illustrate the manner in which the original method is scaled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gLlwfhHEF2fiz8d4w0oW4Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mn2rzR-g9d4ESjazJebZvQ.png" /><figcaption>In-graph replication (left) largely duplicates the computation graph.</figcaption></figure><p>Recognizing the limitations of in-graph replication at scale, between-graph replication aims to be performant with large numbers of nodes. This is achieved by creating a copy of the computation graph on each worker and eliminating the need for the master to host a copy of each worker’s graph. Coordination between these separate graphs is performed through a bit of TensorFlow magic — if two separate nodes allocate a variable on the same TensorFlow device with the same name, those allocations are merged and the variable will share the same backend storage, thus coupling the workers together.</p><p>However, care must be taken to ensure correct device placement. If two workers allocate the variable on different devices, this coupling will not occur. To aid in this process, TensorFlow offers <a href="https://www.tensorflow.org/api_docs/python/tf/train/replica_device_setter"><em>replica_device_setter</em></a>. Provided the individual workers create their graphs in the same sequence, <em>replica_device_setter</em> offers a deterministic method for variable allocation, ensuring the variables reside on the same device. This will be demonstrated in the code below.</p><p>Since between-graph replication is largely creating duplicates of the original graph, most of the relevant changes actually reside in the configuration of the nodes in the cluster. Therefore, the following code snippet will focus on just that. It is important to note that this script would typically be executed on each machine in the cluster but with different command-line arguments. We will walk through the code line-by-line now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/558c4a3f5dce59a918ab8d9f7334e7cb/href">https://medium.com/media/558c4a3f5dce59a918ab8d9f7334e7cb/href</a></iframe><p>The first step in running distributed TensorFlow is to specify the architecture of the cluster using <a href="https://www.tensorflow.org/api_docs/python/tf/train/ClusterSpec"><em>tf.train.ClusterSpec</em></a>. Nodes are typically divided into two roles (or “jobs”): parameter servers (“ps”) which host variables and “workers” which perform the heavy computation. The IP address and port of each node is provided. Next, the script must determine its job type and index within the network; this is typically achieved by passing command-line arguments to the script, which we parse. <em>job_type</em> specifies whether the node is running a ps or worker task, whereas <em>task_idx</em> specifies the node’s index into the ps or worker list. Using this information, a TensorFlow Server is created to allow for connectivity between devices.</p><p>Next, if the node is a parameter server, it simply joins its threads and waits for them to terminate. While it may seem counterintuitive that there is no ps-specific code, the graph elements are actually pushed to it from the workers.</p><p>Conversely, if the device is a worker, we build our model using the <em>replica_device_setter</em> so that parameters are consistently allocated across our ps servers as discussed earlier. These copies will be largely identical to their single-machine counterparts. Finally, we create a <a href="https://www.tensorflow.org/api_docs/python/tf/Session"><em>tf.Session</em></a> and train our model.</p><p>Hopefully, this post has clarified some of the terminology and techniques associated with distributed TensorFlow. In future posts, we will explore this and other topics in more detail, so feel free to check back periodically.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bf94f0205c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>