<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gaurav Singhal on Medium]]></title>
        <description><![CDATA[Stories by Gaurav Singhal on Medium]]></description>
        <link>https://medium.com/@grvsinghal?source=rss-827dbe551f67------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*PSNW2p1_cTj9UiDi.</url>
            <title>Stories by Gaurav Singhal on Medium</title>
            <link>https://medium.com/@grvsinghal?source=rss-827dbe551f67------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:23:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@grvsinghal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Speed up your code using multiprocessing in python]]></title>
            <link>https://medium.com/@grvsinghal/speed-up-your-code-using-multiprocessing-in-python-36e4e703213e?source=rss-827dbe551f67------2</link>
            <guid isPermaLink="false">https://medium.com/p/36e4e703213e</guid>
            <category><![CDATA[multiprocessing]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[windows]]></category>
            <dc:creator><![CDATA[Gaurav Singhal]]></dc:creator>
            <pubDate>Thu, 22 Nov 2018 11:07:36 GMT</pubDate>
            <atom:updated>2021-12-04T03:57:20.309Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous post: <a href="https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac">Multiprocessing in Python on Windows and Jupyter/Ipython — Making it work</a>, I wrote about how to make python multiprocessing work when you are using the devil combination of Jupyter and Windows. In this post, we will see how it actually speed up your code. Here is a brief review of previous post:</p><p><em>Review: We need to define a worker (a function), which we want to execute in parallel, and then we simply use it. Just that worker needs to be in a separate </em><em>.py file if you are on Jupyter. Then to use it on Windows, you need to put the </em><em>pool statement under an </em><em>if clause. the </em><em>pool statement is basically used to assign number of process, as explained in the </em><a href="https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac"><em>previous post</em></a><em>.</em></p><p>Here is a worker which create and sort an array of a given size, which we store in <em>defs.py</em>:</p><pre>import numpy as np</pre><pre>def createandsort (n):<br> rand = np.random.RandomState(42) #Give a seed to reproduce results<br> a = rand.rand(n) #Generate an array of size n<br> return a.sort() #Sort the array</pre><p>Then we just use it:</p><pre>from multiprocessing import Pool<br>from timeit import default_timer as timer<br>import defs</pre><pre>#Create sizes for 3 arrays. <br>sizes = [10**1 for i in range(0,3)] #Size of each array is 10 here.</pre><pre>#Applying the function sequentially<br>tic = timer()<br>[defs.createandsort(size) for size in sizes]<br>tac = timer()<br>print(&quot;time for sequential sorting: &quot;, tac-tic)</pre><pre>#Using multiprocessing<br>if __name__ == &quot;__main__&quot;:<br>    pool = Pool(processes=3)<br>    tic = timer()<br>    pool.map(defs.createandsort,sizes)<br>    tac = timer()<br>    print(&quot;time for parallel sorting: &quot;,tac-tic)</pre><p>If you run the above code, you will see something like following:</p><pre>time for sequential sorting:  0.00013748130595558905<br>time for parallel sorting:  0.6173960894770971</pre><p>What! doing things in parallel are 4000 to 5000 times slower than doing them sequentially. What is wrong here? Actually nothing, the extra time is due to overhead in invoking all the processors, assigning different tasks to them and then gathering the results. Since the array size is only 10, creation and sorting take very less time. So lets try it on bigger arrays.</p><pre>#Create sizes for 3 arrays. <br>sizes = [5 * 10**6 for i in range(0,3)] #Size of each array is 5 million here.</pre><pre>#Applying the function sequentially<br>tic = timer()<br>[defs.createandsort(size) for size in sizes]<br>tac = timer()<br>print(&quot;time for sequential sorting: &quot;, tac-tic)</pre><pre>#Using multiprocessing<br>if __name__ == &quot;__main__&quot;:<br>    pool = Pool(processes=3)<br>    tic = timer()<br>    pool.map(defs.createandsort,sizes)<br>    tac = timer()<br>    print(&quot;time for parallel sorting: &quot;,tac-tic)</pre><p>When each array is of size 5 million, we get the following result:</p><pre>time for sequential sorting:  1.494535938597437<br>time for parallel sorting:  1.2176664572234586</pre><p>So, the parallel version has slightly outperformed the sequential version. What about even bigger arrays, say 100 billion.</p><pre>time for sequential sorting:  36.15379052735625<br>time for parallel sorting:  14.171565365067181</pre><p>So parallel version is almost 3 times faster than the sequential version, something which we were hoping for.</p><p><strong>What Next:</strong></p><p>I hope to write a post with more practical application of multiprocessing.</p><p><strong>Notes:</strong></p><ol><li>If you are using linux, you will not need to define the <em>worker</em> in a separate file and will not need a if __name__ = &#39;__main__&#39;: clause.</li><li>I have used only 3 processors, you can use multiprocessing.cpu_count() to count your number of processors and use them accordingly.</li></ol><p><strong>Become a member:</strong> <a href="https://medium.com/@grvsinghal/membership">https://medium.com/@grvsinghal/membership</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36e4e703213e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multiprocessing in Python on Windows and Jupyter/Ipython — Making it work]]></title>
            <link>https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac?source=rss-827dbe551f67------2</link>
            <guid isPermaLink="false">https://medium.com/p/2714b49d6fac</guid>
            <category><![CDATA[multiprocessing]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Gaurav Singhal]]></dc:creator>
            <pubDate>Wed, 21 Nov 2018 13:26:21 GMT</pubDate>
            <atom:updated>2021-12-04T03:55:08.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>Multiprocessing in Python on Windows and Jupyter/Ipython — Making it work</h3><p><em>This post is most useful if you are using Windows and Jupyter/Ipython, or atleast one of them.</em></p><p>Have you ever come across the situation where you want to speed up your code and were too afraid to try multiprocessing. Or you tried and your fears came true and nothing worked. Or your took another step and found out that windows does not support forking and child processes can’t be distinguished from parent processes, so you need to include an `if__name__ = &#39;__main__&#39; clause and you tried that and it still did not work, then you came across #TextTooDifficultToUnderstand and finally gave up. If all or any of above are true, you can see this post to get going.</p><p><strong>Question</strong>: Why it does not work ?</p><p><strong>Answer</strong>: Both Jupyter and Windows are to blame here, I will not use this space to put details, but at the end you can find few links which explain it better</p><p><strong>Question</strong>: How to make it work?</p><p><strong>Answer</strong>: <em>We just need to do two simple modifications to make it work</em>. But before that, <em>below is how it is done on linux </em>without using Jupyter or Ipython. Probably this is the first thing you came across (or will come across). Basically It consists of two steps: First, create a function, and then use multiple processors to execute the function in parallel.</p><pre>#import Pool<br>from multiprocessing import Pool</pre><pre>#Define a worker — a function which will be executed in parallel<br>def worker(x):<br> return x*x</pre><pre>#Assuming you want to use 3 processors<br>num_processors = 3</pre><pre>#Create a pool of processors<br>p=Pool(processes = num_processors)</pre><pre>#get them to work in parallel<br>output = p.map(worker,[i for i in range(0,3)])</pre><pre>print(output)</pre><p>This code should work perfectly fine on linux, but, If you run it on python shell on Windows (cmd → python), you will get an error like this Can&#39;t get attribute ‘worker&#39; on &lt;module ‘__main__&#39; (built-in)&gt; However if you run it on Jupyter, it will be stuck forever and never complete the processing.</p><p>All you need to do is:</p><ol><li>Define your worker in a separate .py file and import it.</li><li>Add a if __name == &#39;__main__&#39;clause before calling your worker</li></ol><p>Suppose you save the code in workers.py, so it will look like this:</p><pre>def worker(x):<br>    return x*x</pre><p>And just import this file in Jupyter and use workers.worker with an if clause to make it work. So your Jupyter code will look like this:</p><pre>from multiprocessing import Pool<br>import workers</pre><pre>if __name__ ==  &#39;__main__&#39;: <br> num_processors = 3<br> p=Pool(processes = num_processors)<br> output = p.map(workers.worker,[i for i in range(0,3)])<br> print(output)</pre><p>Let me know if this still does not work for you. I have used python 3.6.5.</p><p><strong>What Next:</strong></p><p>In my next post: <a href="https://medium.com/@grvsinghal/speed-up-your-code-using-multiprocessing-in-python-36e4e703213e">Speed up your code using multiprocessing in python</a> , I will show how multiprocessing can actually improve the performance, using a very simple but useful example.</p><p><strong>Details of why simple linux version does not work on Windows/Jupyter and get stuck forever:</strong></p><ol><li>Why multiprocessing does not work in Jupyter or Ipython or any other interactive shell: <a href="https://stackoverflow.com/a/23641560/4613606">https://stackoverflow.com/a/23641560/4613606</a></li><li>Why multiprocessing does not work on Windows without the if clause: <a href="https://stackoverflow.com/questions/20222534/python-multiprocessing-on-windows-if-name-main">https://stackoverflow.com/questions/20222534/python-multiprocessing-on-windows-if-name-main</a></li></ol><p><strong>Become a member: </strong><a href="https://medium.com/@grvsinghal/membership">https://medium.com/@grvsinghal/membership</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2714b49d6fac" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>