<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nima Tajbakhsh on Medium]]></title>
        <description><![CDATA[Stories by Nima Tajbakhsh on Medium]]></description>
        <link>https://medium.com/@nimatajbakhsh?source=rss-c44b1bd1c417------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*wYjeh4QoXuFtoGBD.</url>
            <title>Stories by Nima Tajbakhsh on Medium</title>
            <link>https://medium.com/@nimatajbakhsh?source=rss-c44b1bd1c417------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:59:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nimatajbakhsh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Explaining memory-efficient inference of MobileNet v2.0]]></title>
            <link>https://medium.com/@nimatajbakhsh/explaining-memory-efficient-inference-of-mobilenet-v2-0-10848e2bc622?source=rss-c44b1bd1c417------2</link>
            <guid isPermaLink="false">https://medium.com/p/10848e2bc622</guid>
            <dc:creator><![CDATA[Nima Tajbakhsh]]></dc:creator>
            <pubDate>Sat, 04 Apr 2020 16:13:24 GMT</pubDate>
            <atom:updated>2020-04-05T22:38:27.821Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I first briefly review the linear inverted bottleneck suggested in MobileNet v2 and then explain how it enables memory-efficient inference on a low-end device such as smart phones.</p><p><strong>NOTE</strong>: This is a relatively advance post, which is intended for people who want to gain a deeper understanding of MobileNet v2. For a general overview of the paper, I suggest that you refer to other posts available on medium.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — —</p><h3>Linear Inverted Bottleneck</h3><p>MobileNet v2 paper proposes a new convolution block called linear inverted residual bottleneck, consisting of three parts:</p><ul><li><strong><em>A</em></strong>: point-wise convolution to expand features. This is simply a 1x1 conv with the following effect: HWC1 → HWC2</li><li><strong><em>N</em></strong>: depth-wise separable convolution. This is a 3x3 group convolution where the number of groups is set to C2: HWC2 → HWC2. Depth-wise convolution significantly reduces the number of weights.</li><li><strong><em>B</em></strong>: point-wise convolution to reduce features. This is simply a 1x1 conv with the following effect: HWC2 → HWC3</li></ul><p>Here is the corresponding Pytorch code snippe:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*0tDmL72h2KGzyrGGadIdkg.png" /></figure><p>It is called inverted bottleneck because the features are first expanded (via <strong>A</strong>) and then contracted (via <strong>B</strong>), which is the opposite of a typical bottleneck block. It is also called linear because the last convolution layer (<strong><em>B</em></strong>) uses a linear activation function. The intuition and reasoning behind this new convolution block is beyond the scope of this article — readers are referred to other <a href="https://machinethink.net/blog/mobilenet-v2/">posts</a> for a detailed explanation of such design choices.</p><h3>Memory-Efficient Inference</h3><p>One key advantage of linear inverted bottleneck is memory-efficient inference. The figure below qoutes what the authors state about the efficient inference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*DeJFyoUndMazDmYKL1BDCQ.png" /></figure><p>In what follows, I explain how we can derive the equation above. Given a stack of feature maps <strong><em>I</em></strong>, linear inverted bottleneck applies the following operations, where * denotes convolution:</p><p><strong><em>O1</em></strong> = <strong>A</strong> * <strong><em>I</em></strong></p><p><strong><em>O2 </em></strong>= <strong>N</strong> * <strong><em>O1</em></strong></p><p><strong><em>O3</em></strong> = <strong>B</strong> * <strong><em>O2</em></strong></p><p>For memory-efficient inference, the authors propose to compute the inner tensors (<strong><em>O1</em></strong> and <strong><em>O2</em></strong>) one channel at a time, as opposed to full materialization of these tensors. Recall that <strong><em>O1</em></strong> and <strong><em>O2</em></strong> are tensor with expanded features; and thus, they are expensive to hold in memory. Let’s see how this can be done!</p><ul><li>Step 1. apply the first 1x1 kernel of <strong>A</strong> on <strong><em>I </em></strong>to generate<strong><em> </em></strong>the first feature map of <strong><em>O1</em></strong>, denoted as <strong><em>O1</em></strong>[:,:,1]</li><li>Step 2. apply <strong>N</strong> to <strong><em>O1</em></strong>[:,:,1] to generate <strong><em>O2</em></strong>[:,:,1]. This operation is possible because <strong><em>N</em></strong> is a channel-wise function, that is, <strong><em>O2</em></strong>[:,:,1] depends only on <strong><em>O1</em></strong>[:,:,1].</li><li>Step3. Computing <strong><em>O3[:,:,1], </em></strong>however, requires all feature maps in <strong><em>O2 </em></strong>to be available<strong><em>, </em></strong>because<strong><em> </em>B</strong> is a cross-channel function! Sadly, all we have so far is only<strong><em> O2[:,:,1]</em></strong>. How can we get around this? Let’s write out <strong><em>O3</em></strong> = <strong>B</strong> * <strong><em>O2 </em></strong>to see<strong><em> </em></strong>how each feature map of <strong><em>O3</em></strong> is computed:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*Vc-6SaBRpgbD_0XF6jK9Jg.png" /></figure><p>Recall that <strong>B</strong> performs a 1x1 convolution. Here I use <strong>B</strong> to refer to its 1x1xC2xC3 weight matrix, where C2 and C3 are the number of input and output channels. As can be seen, with <strong><em>O2</em></strong>[:,:,1], we <em>are</em> able to compute all terms in the first column (highlighted in green). So, we compute the first column and put the results in tensor <strong><em>T </em></strong>of size HWC3. At this point, we no longer need <strong><em>O1</em></strong>[:,:,1] and <strong><em>O2</em></strong>[:,:,1]; thus, their allocated memory can be released.</p><p>We now go back to step 1 and next time we end up in step 3, we compute the second column (highlighted in yellow) and then simply add the resulting tensor to <strong><em>T</em></strong>. The accumulated tensor <strong><em>T</em></strong> at step <strong><em>C3</em></strong> will be <strong><em>O3</em></strong>. Basically, we have managed to compute <strong><em>O3</em></strong> without saving <strong><em>O1</em></strong> and <strong><em>O2</em></strong> in memory. I hope this explanation has clarified the equation quoted from the paper.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10848e2bc622" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Multi-threaded Custom Data Pipelines for Tensorflow]]></title>
            <link>https://medium.com/@nimatajbakhsh/building-multi-threaded-custom-data-pipelines-for-tensorflow-f76e9b1a32f5?source=rss-c44b1bd1c417------2</link>
            <guid isPermaLink="false">https://medium.com/p/f76e9b1a32f5</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[generator]]></category>
            <category><![CDATA[multithreading]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Nima Tajbakhsh]]></dc:creator>
            <pubDate>Wed, 08 May 2019 00:55:50 GMT</pubDate>
            <atom:updated>2019-05-20T22:29:17.215Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Multi-threaded Custom Data Pipelines for Tensorflow v1.12+</h3><p><em>Written by </em><a href="https://medium.com/u/c44b1bd1c417"><em>Nima Tajbakhsh</em></a><em> and </em><a href="https://medium.com/u/a016e48f2503"><em>Jeff Chiang</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wdiqf985aBLuptPzVx_oFA.jpeg" /><figcaption><a href="https://www.bdlaw.com/pipeline-safety-resource-center/">source: https://www.bdlaw.com/pipeline-safety-resource-center/</a></figcaption></figure><p>You are dealing with problems where data, fortunately or unfortunately, is not available in the typical 2D image format. For instance, your data may consist of 3D medical scans, which are available in DICOM format. As such, <br>you cannot benefit from the standard Tensorflow (TF) data i/o functions (e.g. from_csv, from_pandas, etc) or data pre-processing functions (e.g. in tf.image) available for regular 2D images. Facing all these limitations, you have written your own customized, super fancy data pipeline in python, which interfaces with your model through the (soon obsolete) feed_dict mechanism. Your data pipeline works just fine, but you have always seen sub-optimal GPU usage when running <em>nvidia-smi</em>, thinking to yourself the data pipeline might not be optimized.</p><p>You have now heard that <a href="https://www.tensorflow.org/alpha/guide/migration_guide">Tensorflow 2.0</a> is coming, and with it a potentially massive refactoring headache. Among many suggested changes is the following: “use tf.data datasets for data input”. You are so tempted to give it a try, seeing for yourself whether your data pipeline can get any further optimized. Having tons of respect for yourself as an independent problem-solver, you opened the TF documentation, passionately scrolled up and down the pages for hours, and soon become frustrated from being unable to make sense of this supposedly comprehensive documentation. You think to yourself, “How come I cannot make much sense out of it?! Have I lost my edge or is this documentation just one of a kind? Sigh! Only if I could have some proper pointers on how to use this beautiful module of TF.” Well, if this is you, stop fretting about it, because you have just found what you were looking for: a simple, easy-to-follow tutorial on how to make a fast, multi-threaded data pipeline based off of a python generator. And the best part is, <strong>what you learn here will be applicable to both TF1.12+ and TF 2.0.</strong></p><h3>TL; DR</h3><p>1- Pre-processing operations in pure Python (or numpy, etc) can be integrated with tf.data using <em>from_generator</em><br> <br>2- Blindly wrapping your generator using <em>from_generator</em> is single-threaded, and suboptimal. Multi-threading exists in tf.data.<br> <br> 3- Additional speed-up can be gained by using prefetch() and separating time-consuming operations</p><p>4- Source codes for this tutorial are available <a href="https://github.com/ntajbakhsh/fast_tf_data">here</a>.</p><h3>Structure of this tutorial</h3><p>This tutorial explains tf.data through three easy-to-understand examples. Example 1 is a must-read, walking you through the most important concepts and pitfalls when using tf.data. You will learn how to write single-threaded and multi-threaded data pipelines as well as how to use pre-fetching to turn undesirable periodic delays into opportunities to keep your data queue full. In Example 2, you will learn how to set up your data pipeline using a python class and how to avoid some common pitfalls. Example 3 teaches you how you can break a data pipeline into two smaller pipelines, further accelerating the data preparation process for your models.</p><h3>Example 1</h3><p>We have a python-based data generator, which takes 0.3 seconds to prepare data (for example, loading a large numpy array from disk and applying some data augmentation) and then we have this sophisticated TF operation (for example, a training step for an object detection model), which processes the input data in 0.1 seconds. We can simulate this scenario with the following data generator and TF operation:</p><pre><em>import </em>tensorflow <em>as </em>tf<br><em>from </em>time <em>import </em>sleep<br><em>from </em>time <em>import </em>time<br><br><em># data generator<br>def </em>py_gen(gen_name):<br>    gen_name = gen_name.decode(&#39;utf-8&#39;)<br>    <em>for </em>num <em>in </em>range(20):<br>        sleep(0.3)<br>        <em>yield </em>&#39;{} yields {}&#39;.format(gen_name, num)<br><br><br><em># model operation<br>def </em>model(data):<br>    sleep(0.1)</pre><p>In this example, the generator takes as the input a name and then yields a string at each iteration. A pause of 0.3 seconds is included in the body of the generator to simulate a set of time-consuming operations, while the sophisticated TF operation is simulated as a pause of 0.1 seconds.</p><p>We start this example by writing a single-threaded data pipeline using tf.data. We then show how adding a dedicated data thread and using a prefetch mechanism can optimize this pipeline. Finally, we go over a multi-threaded solution to this example.</p><p><strong>Single-threaded solution</strong><br>Using <em>from_generator</em>, we can convert our generator to a basic data pipeline.</p><pre>Dataset = tf.data.Dataset<br>name = &#39;Gen_0&#39;<br>ds = Dataset.from_generator(py_gen,<br>                            output_types=(tf.string),<br>                            args=(name,))<br>data_tf = ds.make_one_shot_iterator().get_next()</pre><p>We have simply defined a name for our generator and then passed the data generator, py_gen, to the Dataset object of TF, which uses a data queue in its back-end. We then create an <a href="https://www.tensorflow.org/guide/datasets#creating_an_iterator">iterator</a> that runs through the queue. Each time the iterator moves forward, a string is yielded.</p><p><strong>Note: </strong>when passing the python generator to the Dataset object, you must specify the types of the tensors that your python generator returns. In this example, we return a string; thus, we have specified tf.string as the output type.</p><p><strong>Note:</strong> we need to use “args” to pass the input arguments (if any) to our python generator. Beware that the code will crash if we omit the comma in the tuple provided for “args”.</p><p><strong>Note:</strong> TF returns string objects that are encoded as Bytes, so we need to decode them into a string. This is done using decode(‘utf-8’).</p><p><strong>Note:</strong> We know that TF 2.0 discourages the use of tf.Session(), but the timing differences are the same in both TF1.12+ and TF 2.0.</p><p>Let’s run the session and see the output:</p><pre><em>def </em>run_session(data_tf):<br>    <em>with </em>tf.Session() <em>as </em>sess:<br>        <em>while True</em>:<br>            <em>try</em>:<br>                t1 = time()<br>                data_py = sess.run(data_tf)<br>                t2 = time()<br>                t = t2 - t1<br>                model(data_tf)<br>                msg = &#39;elapsed time: {:.3f}, {}&#39;.format(t, data_py.decode(&#39;utf-8&#39;))<br>                print(msg)<br>            <em>except </em>tf.errors.OutOfRangeError:<br>                print(&#39;data generator(s) are exhausted&#39;)<br>                <em>break</em></pre><pre>elapsed time: 0.320, Gen_0 yields 0<br>elapsed time: 0.302, Gen_0 yields 1<br>elapsed time: 0.302, Gen_0 yields 2<br>elapsed time: 0.302, Gen_0 yields 3<br>elapsed time: 0.302, Gen_0 yields 4</pre><p>We’ve converted our data pipeline to tf.data, but wait a minute! In this example, the model operation takes 0.1 seconds to finish, so if the data generator were working in the background, it would take only 0.2 (=0.3–0.1) seconds to generate the next input. This observation suggests that the data generator does not take advantage of 0.1-second delay caused by the model. In other words, we could achieve faster performance if we loaded the next data while we perform the model operations on the current data. In fact, in this case the same thread prepares the data and also runs the TF operation. Thus, the preparation of the next input is postponed until after the TF operation has ended. If the data generator owned its own thread, the data preparation and TF operation could run simultaneously, reducing the wait time for the TF operation from 0.3 to 0.2 seconds. So let’s assign the data generator a dedicated thread. Actually, the fix is quite simple:</p><pre>Dataset = tf.data.Dataset<br>name = &#39;Gen_0&#39;<br>ds = Dataset.from_generator(py_gen,<br>                            output_types=(tf.string),<br>                            args=(name,))<br>ds = ds.map(<em>lambda </em>x: x, num_parallel_calls=1)<br>data_tf = ds.make_one_shot_iterator().get_next()</pre><p>The difference with the previous solution is in Line 6, where I apply an identity transformation to each element in the dataset. Of course, it is not the identity transformation that does the job, rather, <em>num_parallel_calls=1</em>, which assigns one dedicated thread to the data generator. To be frank, we are somewhat abusing the parallel feature of the map function, but it serves the purpose. Let’s see the output:</p><pre>elapsed time: 0.322, Gen_0 yields 0<br>elapsed time: 0.201, Gen_0 yields 1<br>elapsed time: 0.201, Gen_0 yields 2<br>elapsed time: 0.201, Gen_0 yields 3<br>elapsed time: 0.201, Gen_0 yields 4</pre><p><strong>Note:</strong> By default, <em>num_parallel_calls</em> is set to 0, instructing the data pipeline to use the existing thread. <em>num_parallel_calls=1</em> actually instructs tf.data to create a new thread for this operation.</p><p>The TF operation now has to wait only 0.2 seconds for the input data, which is what we wanted to achieve. Of course, the wait time is still 0.3 seconds for the very first input data, because the TF operation has not been executed yet; and thus, the 0.1-second delay cannot be taken advantage of.</p><p>Before we move on to a multi-threaded solution, let’s inject a one-time delay to our TF session. As seen below, we now have a delay of 3 seconds at the onset of iteration 4, which gives the dedicated data thread ample amount of time to create the next 10 input data for the model operation. Recall that the data thread needs 0.3 seconds to prepare each input data.</p><pre><em>def </em>run_session_w_delay(data_tf):<br>    <em>with </em>tf.Session() <em>as </em>sess:<br>        counter = 1<br>        <em>while True</em>:<br>            <em>try</em>:<br>                <em>if </em>counter == 4:<br>                    sleep(3)<br>                t1 = time()<br>                data_py = sess.run(data_tf)<br>                t2 = time()<br>                t = t2 - t1<br>                model(data_tf)<br>                msg = &#39;elapsed time: {:.3f}, {}&#39;.format(t, data_py.decode(&#39;utf-8&#39;))<br>                print(msg)<br>                counter += 1<br>            <em>except </em>tf.errors.OutOfRangeError:<br>                print(&#39;data generator(s) are exhausted&#39;)<br>                <em>break</em></pre><p>Let’s run the session and see the output:</p><pre>elapsed time: 0.322, Gen_0 yields 0<br>elapsed time: 0.201, Gen_0 yields 1<br>elapsed time: 0.201, Gen_0 yields 2<br>elapsed time: 0.001, Gen_0 yields 3<br>elapsed time: 0.201, Gen_0 yields 4<br>elapsed time: 0.201, Gen_0 yields 5<br>elapsed time: 0.201, Gen_0 yields 6<br>elapsed time: 0.201, Gen_0 yields 7<br>elapsed time: 0.201, Gen_0 yields 8<br>elapsed time: 0.201, Gen_0 yields 9</pre><p>Iterations 1 to 3 are just the same as before. As expected, iteration 4 has no delay thanks to the 3-second delay that we injected. However, surprisingly, we go back to having a delay of 0.2 seconds for iteration 5 and onward! This is counter-intuitive because the data thread has had enough time to prepare the next 10 input data, then why has it created only the next input?! Well, this may be a design consideration to keep memory usage limited especially when the input data is memory-intensive (let’s say a 3D scan). So, what’s the solution? Easy! Just have the data thread prefetch the input data, which can be done by adding one line of code, as follows:</p><pre>ds = Dataset.from_generator(py_gen,<br>                            output_types=(tf.string),<br>                            args=(name,))<br>ds = ds.map(<em>lambda </em>x: x, num_parallel_calls=1)<br>ds = ds.prefetch(buffer_size=10)<br>data_tf = ds.make_one_shot_iterator().get_next()</pre><p>Buy setting <em>buffer_size=10</em>, we instruct the data thread to prepare and hold 10 input data in the queue. If we run the code once again, we see no delay after iteration 4.</p><pre>elapsed time: 0.324, Gen_0 yields 0<br>elapsed time: 0.201, Gen_0 yields 1<br>elapsed time: 0.201, Gen_0 yields 2<br>elapsed time: 0.001, Gen_0 yields 3<br>elapsed time: 0.001, Gen_0 yields 4<br>elapsed time: 0.001, Gen_0 yields 5<br>elapsed time: 0.001, Gen_0 yields 6<br>elapsed time: 0.001, Gen_0 yields 7<br>elapsed time: 0.001, Gen_0 yields 8<br>elapsed time: 0.001, Gen_0 yields 9</pre><p>Of course, this no-delay behavior won’t last long, because the model runs faster than the data generator in this example. So, after a certain number of iterations, we once again observe a delay of 0.2 seconds prior to each model operation.</p><p>You may be wondering what the point of using data prefetching is, if it is only going to be effective for a few iterations? In my opinion, data prefetching begins to shine when we have a periodic delay in our TF session, as opposed to a 1-time delay in this example. A periodic delay could be some post-processing or data IO that can take place after every several iterations.</p><p><strong>Multi-threaded solution</strong><br> A common approach to creating a multi-threaded pipeline is to instantiate several “replicas” of the data generator and assign them to different threads. It turns out the <em>interleave</em> function of TF Dataset mimics this behavior. We can use it to create several instances of our data generator, allowing us to pull data from all instances in parallel, and then interleave the pulled data into the final data queue. To take advantage of <em>interleave</em>, you only need to change the following lines of code:</p><pre>Dataset = tf.data.Dataset<br>ds = Dataset.from_tensor_slices([&#39;Gen_0&#39;, &#39;Gen_1&#39;, &#39;Gen_2&#39;])<br>ds = ds.interleave(<em>lambda </em>x: Dataset.from_generator(py_gen, output_types=(tf.string), args=(x,)),<br>                   cycle_length=3,<br>                   block_length=1,<br>                   num_parallel_calls=3)<br>data_tf = ds.make_one_shot_iterator().get_next()</pre><p>The multi-threaded pipeline removes the wait time for the TF operation. Here is the output:</p><pre>elapsed time: 0.327, Gen_0 yields 0<br>elapsed time: 0.001, Gen_1 yields 0<br>elapsed time: 0.001, Gen_2 yields 0<br>elapsed time: 0.001, Gen_0 yields 1<br>elapsed time: 0.001, Gen_1 yields 1<br>elapsed time: 0.001, Gen_2 yields 1<br>elapsed time: 0.001, Gen_0 yields 2<br>elapsed time: 0.001, Gen_1 yields 2<br>elapsed time: 0.001, Gen_2 yields 2<br>elapsed time: 0.001, Gen_0 yields 3<br>elapsed time: 0.001, Gen_1 yields 3<br>elapsed time: 0.001, Gen_2 yields 3<br>elapsed time: 0.001, Gen_0 yields 4<br>elapsed time: 0.001, Gen_1 yields 4<br>elapsed time: 0.001, Gen_2 yields 4</pre><p>You might be scratching your head thinking what the hell just happened. I agree — these few lines of code are a little too hard to digest particularly for someone who is new to the Dataset module of TF. Well, I presented the solution right away to make it easy for those of you who are here looking for a quick answer. We now go through the code above line-by-line.</p><p>It all starts with creating a base dataset that contains, in each element, the arguments needed to initialize our data generator. Here, I intend to have 3 instances of data generator and each generator needs a mandatory string argument. Hence, I have used <em>from_tensor_slices</em> to generate a dataset with 3 elements where each element is a string. Obviously, if the generator needed 2 input arguments, I would have created a dataset where each element is a tuple with 2 values. Once the base dataset is constructed, I call the <em>interleave</em> method, which applies a same function to each element in the dataset. As you can guess this function is nothing but the same old <em>from_generator</em>, which takes as input the current string element, <em>x</em>, and then creates a new dataset object via our <em>py_gen</em>. By using <em>from_tensor_slices</em> and <em>interleave </em>methods, I have asked TF to generate 3 datasets each fed by 1 instance of our data generator.</p><p>Now, let’s merge these 3 datasets into 1 final dataset. Fortunately, we do not need to write extra codes for this purpose, as the interleave method accepts two additional arguments called <em>cycle_length</em> and <em>block_length</em>, which allow us to combine the resulting datasets in a flexible manner:</p><ul><li>if <em>cycle_length</em> is set to 1, the merged dataset begins with elements of dataset 1, and then elements of dataset 2, and then elements of dataset 3.</li><li>if <em>cycle_length</em> is set to 2, then the merged dataset will have <em>block_length</em> elements of dataset 1, followed by <em>block_length</em> of dataset 2, and then <em>block_length</em> elements of dataset 1, followed by <em>block_length</em> of dataset 2, and it continues until datasets 1 and 2 are exhausted. Next, elements of dataset 3 are appended to the merged dataset.</li><li>if <em>cycle_length</em> is set to 3, then the merged dataset will have <em>block_length</em> elements of dataset 1, followed by <em>block_length</em> of dataset 2, followed by <em>block_length</em> elements of dataset 3, and then back to dataset 1 and this loop continues until all 3 dataset are exhausted.</li></ul><p>So, you are thinking how to set these 2 parameters? Easy!<br>Set the <em>cycle_length</em> equal to the number of datasets you have created and set <em>block_length</em> to 1. By doing so, dataset 1 will have ample amount of time to prepare the next element while our TF operation consumes elements from datasets 2 and 3.</p><p>One last thing, TF by default assigns the same thread to all 3 datasets, meaning no performance improvement whatsoever. You need to manually set <em>num_parallel_calls</em> to 3, which assigns a separate thread to each instance of our data generator, resulting in a true multi-threaded data pipeline.</p><p><strong>Note:</strong> We discussed the necessity of having a base dataset as a means of providing the input arguments for multiple instances of our data generator. In the example above, the base datasets had 3 strings in it. But what if the data data generator needs no input arguments? Well, we still need to have a base dummy dataset with as many elements as the number of generator instances we wish to create. The content of each element does not matter though — it can be an int, float or string. Here is an example where the data generator needs no input argument, but we still need to use a base dummy dataset:</p><pre><em># data generator<br>def </em>py_gen():<br>    <em>for </em>num <em>in </em>range(5):<br>        sleep(0.3)<br>        <em>yield </em>&#39;{} yields {}&#39;.format(&#39;no name&#39;, num)</pre><pre><em># set up tf generator<br></em>Dataset = tf.data.Dataset<br>ds = Dataset.from_tensor_slices([&#39;xxx&#39;, &#39;xxx&#39;, &#39;xxx&#39;])<br>ds = ds.interleave(<em>lambda </em>x: Dataset.from_generator(py_gen, output_types=(tf.string)),<br>                   cycle_length=3,<br>                   block_length=1,<br>                   num_parallel_calls=3)<br>data_tf = ds.make_one_shot_iterator().get_next()</pre><h3>Example 2</h3><p>It is often desirable to have the data generator as an instance method of a python class. By doing so, different instances of our data generators can have access to class-level attributes. In fact, instead of passing a bunch of common arguments to your generators, you can define the arguments as class attributes, and let the generator instances access them during the run-time. For instance, I have put together a data generator class that allows me to create the generator name on-the-fly by incrementally updating a <em>counter</em> variable, which I have defined as a class attribute as follows:</p><pre><em>class </em>DataGen():<br>    counter = 0<br><br>    <em>def __init__</em>(<em>self</em>):<br>        <em>self</em>.gen_num = DataGen.counter<br>        DataGen.counter += 1<br><br>    <em>def </em>py_gen(<em>self</em>, gen_name):<br>        gen_name = gen_name.decode(&#39;utf8&#39;) + &#39;_&#39; + str(<em>self</em>.gen_num)<br>        <em>for </em>num <em>in </em>range(5):<br>            sleep(0.3)<br>            <em>yield </em>&#39;{} yields {}&#39;.format(gen_name, num)</pre><p>Since I have changed the interface to our data generator, I also need to change what I pass to <em>from_generator</em> as follows:</p><pre>Dataset = tf.data.Dataset<br>dummy_ds = Dataset.from_tensor_slices([&#39;Gen&#39;, &#39;Gen&#39;, &#39;Gen&#39;])<br>dummy_ds = dummy_ds.interleave(<em>lambda </em>x: Dataset.from_generator(DataGen().py_gen, output_types=(tf.string), args=(x,)),<br>                               cycle_length=3,<br>                               block_length=1,<br>                               num_parallel_calls=3)<br>data_tf = dummy_ds.make_one_shot_iterator().get_next()</pre><p>Now the desired output should look like “Gen_0 yields …” followed by “Gen_1 yields …” and so on, but let’s run the code and see the output:</p><pre>elapsed time: 0.329, Gen_0 yields 0<br>elapsed time: 0.001, Gen_0 yields 0<br>elapsed time: 0.001, Gen_0 yields 0<br>elapsed time: 0.001, Gen_0 yields 1<br>elapsed time: 0.001, Gen_0 yields 1<br>elapsed time: 0.001, Gen_0 yields 1<br>elapsed time: 0.001, Gen_0 yields 2<br>elapsed time: 0.001, Gen_0 yields 2<br>elapsed time: 0.001, Gen_0 yields 2<br>elapsed time: 0.001, Gen_0 yields 3<br>elapsed time: 0.001, Gen_0 yields 3<br>elapsed time: 0.001, Gen_0 yields 3<br>elapsed time: 0.001, Gen_0 yields 4<br>elapsed time: 0.001, Gen_0 yields 4<br>elapsed time: 0.001, Gen_0 yields 4</pre><p>Okay, we have a big problem! The GEN ID always remains the same (Gen_0). I must confess this output was quite puzzling to me as well: why the heck is the GEN ID not updated? It turns out that python evaluates the input argument to <em>from_generator</em> only once, as such, the <em>counter</em> is updated only once. The fix is to force python into evaluating the input argument for each data element passed, which can be done by placing the input argument into a lambda function. Here is the fix and the resulting output:</p><pre>Dataset = tf.data.Dataset<br>dummy_ds = Dataset.from_tensor_slices([&#39;Gen&#39;, &#39;Gen&#39;, &#39;Gen&#39;])<br>dummy_ds = dummy_ds.interleave(<br>    <em>lambda </em>x: Dataset.from_generator(<em>lambda </em>x: DataGen().py_gen(x), output_types=(tf.string), args=(x,)),<br>    cycle_length=3,<br>    block_length=1,<br>    num_parallel_calls=3)<br>data_tf = dummy_ds.make_one_shot_iterator().get_next()</pre><pre>elapsed time: 0.327, Gen_0 yields 0   <br>elapsed time: 0.001, Gen_1 yields 0   <br>elapsed time: 0.001, Gen_2 yields 0   <br>elapsed time: 0.001, Gen_0 yields 1   <br>elapsed time: 0.001, Gen_1 yields 1   <br>elapsed time: 0.001, Gen_2 yields 1   <br>elapsed time: 0.001, Gen_0 yields 2   <br>elapsed time: 0.001, Gen_1 yields 2   <br>elapsed time: 0.001, Gen_2 yields 2   <br>elapsed time: 0.001, Gen_0 yields 3   <br>elapsed time: 0.001, Gen_1 yields 3   <br>elapsed time: 0.001, Gen_2 yields 3   <br>elapsed time: 0.001, Gen_0 yields 4   <br>elapsed time: 0.001, Gen_1 yields 4   <br>elapsed time: 0.001, Gen_2 yields 4</pre><p>In this example, the bug above resulted in only some error in the print messages, which is not a big deal. However, this bug could have a devastating effect if the generator required vital data from the class attributes.</p><h3>Example 3</h3><p>Now, let’s consider a 2-stage data generator where stage 1 reads a large 3D image (e.g., a chest CT scan), and stage 2 extracts a number of local cubes from random locations in this 3D image. Stage 1 is slow, taking 1.5 seconds to load one 3D image, but stage 2 is much faster, taking only 0.3 seconds to return a local cube. We simulate this problem by defining the following data generator:</p><pre><em># data generator<br>def </em>py_gen(num):<br>    <em>for </em>_ <em>in </em>range(3):<br>        tensor = np.arange(num * 10, (num + 1) * 10)<br>        sleep(1.5)<br>        <em>for </em>x <em>in </em>range(5):<br>            sleep(0.3)<br>            <em>yield </em>tensor[x * 2: (x + 1) * 2]</pre><p>where I have used 1D tensors instead of 3D tensors for code brevity. Since we have changed the data generator, some minor modifications in the TF code are necessary:</p><pre><em># set up tf generator<br></em>Dataset = tf.data.Dataset<br>ds = Dataset.from_tensor_slices([0, 1, 2])<br>ds = ds.interleave(<em>lambda </em>x: Dataset.from_generator(py_gen, output_types=(tf.float32), args=(x,)),<br>                               cycle_length=3,<br>                               block_length=1,<br>                               num_parallel_calls=3)<br>data_tf = ds.make_one_shot_iterator().get_next()</pre><p>Running the session gives the following output:</p><pre>elapsed time: 1.828, [0. 1.]<br>elapsed time: 0.001, [10. 11.]<br>elapsed time: 0.001, [20. 21.]<br>elapsed time: 0.001, [2. 3.]<br>elapsed time: 0.001, [12. 13.]<br>elapsed time: 0.001, [22. 23.]<br>elapsed time: 0.001, [4. 5.]<br>elapsed time: 0.001, [14. 15.]<br>elapsed time: 0.001, [24. 25.]<br>elapsed time: 0.001, [6. 7.]<br>elapsed time: 0.001, [16. 17.]<br>elapsed time: 0.001, [26. 27.]<br>elapsed time: 0.001, [8. 9.]<br>elapsed time: 0.001, [18. 19.]<br>elapsed time: 0.001, [28. 29.]<br>elapsed time: 1.498, [0. 1.]<br>elapsed time: 0.001, [10. 11.]<br>elapsed time: 0.001, [20. 21.]<br>elapsed time: 0.001, [2. 3.]<br>elapsed time: 0.001, [12. 13.]<br>elapsed time: 0.001, [22. 23.]<br>elapsed time: 0.001, [4. 5.]<br>elapsed time: 0.001, [14. 15.]<br>elapsed time: 0.001, [24. 25.]<br>elapsed time: 0.001, [6. 7.]<br>elapsed time: 0.001, [16. 17.]<br>elapsed time: 0.001, [26. 27.]<br>elapsed time: 0.001, [8. 9.]<br>elapsed time: 0.001, [18. 19.]<br>elapsed time: 0.001, [28. 29.]<br>elapsed time: 1.499, [0. 1.]<br>elapsed time: 0.001, [10. 11.]<br>elapsed time: 0.001, [20. 21.]<br>elapsed time: 0.001, [2. 3.]<br>elapsed time: 0.001, [12. 13.]<br>elapsed time: 0.001, [22. 23.]<br>elapsed time: 0.001, [4. 5.]<br>elapsed time: 0.001, [14. 15.]<br>elapsed time: 0.001, [24. 25.]<br>elapsed time: 0.001, [6. 7.]<br>elapsed time: 0.001, [16. 17.]<br>elapsed time: 0.001, [26. 27.]<br>elapsed time: 0.001, [8. 9.]<br>elapsed time: 0.001, [18. 19.]<br>elapsed time: 0.001, [28. 29.]</pre><p>Not too bad except that we have a relatively large lag in our data generator after every 15 iterations. This is because we assign 1 thread to each data generator: that same thread is responsible for creating a tensor (1.5 seconds) and then performing tensor slicing (0.3 seconds for each slice). As such, we will experience a delay when all slices are extracted from the current tensor and the next tensor for slicing needs to be created. You may ask what about the other threads? Can they not compensate for this lag? The answer is that since <em>cycle_length</em> is set to 3, all 3 threads reach stage 1 at the same time, and while stuck there, no tensor slicing can be performed for the subsequent TF operation.</p><p>One fix would be to break the data generator into two sub-generators, each covering one stage of data generation. We can then assign a dedicated thread to each sub-generator and take advantage of asynchronous processing. By doing so, some threads will create the new tensors, while other threads will slice the current tensor. Here are the new data generators:</p><pre><em>def </em>py_gen_p1(num):<br>    <em>for </em>_ <em>in </em>range(3):<br>        sleep(1.5)<br>        <em>yield </em>np.arange(num * 10, (num+1) * 10)<br><br><em>def </em>py_gen_p2(tensor):<br>    <em>for </em>x <em>in </em>range(5):<br>        sleep(0.3)<br>        <em>yield </em>tensor[x * 2: (x + 1) * 2]</pre><p>And here is how we feed the two generators into the Dataset module of TF:</p><pre>Dataset = tf.data.Dataset<br>ds = Dataset.from_tensor_slices([0, 1, 2])<br>ds = ds.interleave(<em>lambda </em>x: Dataset.from_generator(py_gen_p1, output_types=(tf.float32), args=(x,)),<br>                               cycle_length=3,<br>                               block_length=1,<br>                               num_parallel_calls=3)<br>ds = ds.interleave(<em>lambda </em>x: Dataset.from_generator(py_gen_p2, output_types=(tf.float32), args=(x,)),<br>                               cycle_length=3,<br>                               block_length=1,<br>                               num_parallel_calls=3)<br>data_tf = ds.make_one_shot_iterator().get_next()</pre><p>Running the new multi-threaded solution outputs the following:</p><pre>elapsed time: 1.846, [0. 1.]<br>elapsed time: 0.001, [10. 11.]<br>elapsed time: 0.001, [20. 21.]<br>elapsed time: 0.001, [2. 3.]<br>elapsed time: 0.001, [12. 13.]<br>elapsed time: 0.001, [22. 23.]<br>elapsed time: 0.001, [4. 5.]<br>elapsed time: 0.001, [14. 15.]<br>elapsed time: 0.001, [24. 25.]<br>elapsed time: 0.001, [6. 7.]<br>elapsed time: 0.001, [16. 17.]<br>elapsed time: 0.001, [26. 27.]<br>elapsed time: 0.001, [8. 9.]<br>elapsed time: 0.001, [18. 19.]<br>elapsed time: 0.001, [28. 29.]<br>elapsed time: 0.306, [0. 1.]<br>elapsed time: 0.001, [10. 11.]<br>elapsed time: 0.001, [20. 21.]<br>elapsed time: 0.001, [2. 3.]<br>elapsed time: 0.001, [12. 13.]<br>elapsed time: 0.001, [22. 23.]<br>elapsed time: 0.001, [4. 5.]<br>elapsed time: 0.001, [14. 15.]<br>elapsed time: 0.001, [24. 25.]<br>elapsed time: 0.001, [6. 7.]<br>elapsed time: 0.001, [16. 17.]<br>elapsed time: 0.001, [26. 27.]<br>elapsed time: 0.001, [8. 9.]<br>elapsed time: 0.001, [18. 19.]<br>elapsed time: 0.001, [28. 29.]<br>elapsed time: 0.305, [0. 1.]<br>elapsed time: 0.001, [10. 11.]<br>elapsed time: 0.001, [20. 21.]<br>elapsed time: 0.001, [2. 3.]<br>elapsed time: 0.001, [12. 13.]<br>elapsed time: 0.001, [22. 23.]<br>elapsed time: 0.001, [4. 5.]<br>elapsed time: 0.001, [14. 15.]<br>elapsed time: 0.001, [24. 25.]<br>elapsed time: 0.001, [6. 7.]<br>elapsed time: 0.001, [16. 17.]<br>elapsed time: 0.001, [26. 27.]<br>elapsed time: 0.001, [8. 9.]<br>elapsed time: 0.001, [18. 19.]<br>elapsed time: 0.001, [28. 29.]</pre><p>As seen, using a dedicated thread for each stage of the data generator reduces the delay from 1.5 seconds to only 0.3 seconds.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f76e9b1a32f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>