<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[NaIA: Machine Learning at the heart of Public Healthcare - Medium]]></title>
        <description><![CDATA[This is the official blog page of the AI team of the Health Technologies Service (Government of Navarra, Spain). Our mission is to build AI-powered software for the Public Heatlhcare. Here we try to make some little contributions to the AI community. We hope they are useful! - Medium]]></description>
        <link>https://medium.com/naia-blog?source=rss----430b56e8eeb8---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>NaIA: Machine Learning at the heart of Public Healthcare - Medium</title>
            <link>https://medium.com/naia-blog?source=rss----430b56e8eeb8---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 20:59:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/naia-blog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Distributed Training across multiple nodes in fastai]]></title>
            <link>https://medium.com/naia-blog/distributed-training-across-multiple-nodes-in-fastai-299c0ef56e2f?source=rss----430b56e8eeb8---4</link>
            <guid isPermaLink="false">https://medium.com/p/299c0ef56e2f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[distributed-computing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Adrian Errea Lopez]]></dc:creator>
            <pubDate>Wed, 27 Apr 2022 10:21:10 GMT</pubDate>
            <atom:updated>2022-04-27T10:21:10.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3MtFp_HCvIdMcA_jejScmw.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/es-es/@tara-winstead?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Tara Winstead</strong></a> at <a href="https://www.pexels.com/es-es/foto/mano-dedo-futuro-robot-8386440/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><p>This is my first post on Medium as part of our AI team called by ourselves <strong>NaIA (Navarra Inteligencia Artificial)</strong>.</p><p>We work for the Health Service to bring the power of AI into the heart of current Hospital Information Systems in Navarra. We have been building AI software since 2019, so we have a lot to share with you.</p><p>As an AI team, we usually make use of our hardware colleagues: <strong>the GPUs</strong>. We see them as colleages because we even named them with real names!</p><p>To get our projects off the ground, we mostly use <a href="https://github.com/fastai/fastai"><strong>fastai</strong></a><strong> </strong>to develop our code and we need to train our Deep Learning algorithms in GPUs. Usually, when the dataset and the model are not very big, we can fit them well in one GPU without any problems.</p><p>Things change when the datasets and models get bigger and bigger and the training phase becomes impossible to carry out or, at least, takes too much time. Something very common in AI solutions for the healthcare, where patients’ studies can be really big!</p><p>At this point, adding another GPU to your workstation could do the trick. For this, fastai (from PyTorch behind the scenes) provides two different modes to work with: <strong>Data-Parallel (DP)</strong> or <strong>Distributed Data-Parallel (DDP)</strong>.</p><p>I am not going to explain the differences here as you can check out the <a href="https://docs.fast.ai/distributed.html">documentation of fastai</a> as well as some great <a href="https://forums.fast.ai/t/distributed-and-parallel-training-explained/73892">threads</a> on the fastai forum.</p><p>However, when we face even larger datasets (NLP or medical images, for instance) or even larger models, <strong>2 or even 3 big GPUs in the same workstation are not enough</strong>. We need more. Maybe this is not an issue for you, because you have an unlimited budget (just kidding) to add more big GPUs to your cloud instance machine. However, if you work on-premise with sensitive data, as we do, and your nodes have limited space for new GPUs, you must add new nodes to your AI cluster.</p><p>And here comes the trouble.</p><p><strong>We need to launch our training phase on multiple GPUs on multiple nodes. In fastai.</strong></p><p>But fastai, in the current version (2.5.6), only allows DDP on a single node. So we must do something to change that.</p><p>Doing the same in plain Pytorch is already solved and you can check out <a href="https://leimao.github.io/blog/PyTorch-Distributed-Training/">this post</a> and <a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html">this post</a> as they explained the process quite well (they were really helpful for writing this post, so I appreciate that).</p><p>I am going to assume that the basic concepts of Distributed Training are known such as <em>nodes</em>, <em>processes</em>, <em>rank_ids </em>or <em>master_IP</em>. Otherwise, this post would be too long. You can have a look at the previous links for a deep dive around those concepts. However, you can continue reading: I am sure you will build some good intuition!</p><p>So, for launching a Distributed Training across multiple nodes in fastai, we had to make some changes to some scripts and environment variables as we are going to see now.</p><h3>Launch.py file</h3><p>First, when launching a DDP in fastai on a single node, we are using the following command: python -m fastai.launch scriptname.py ...args...</p><p>This one launches a module called <a href="https://github.com/fastai/fastai/blob/master/fastai/launch.py"><em>launch.py</em></a><em> </em>within the fastai library with some arguments (<em>scriptname.py and so on</em>).</p><p>To make it work with more nodes, we can replace that file with the following one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdff264df46fc181968809e4355c7784/href">https://medium.com/media/cdff264df46fc181968809e4355c7784/href</a></iframe><p>Notice that, first, I changed the <strong>parameters </strong>of the function. Let’s explain them one by one:</p><ul><li>I added gpus_to_use, which corresponds to the cuda ids I will use in this node. For example, in a node with 3 GPUs, if we just want to use the first and the third, we would set this variable as gpus_to_use 0,2 . I found this useful because sometimes a colleague is using one GPU in one node and I just want to use the other GPU. This allows you to do that. By default, it is set up to <em>all</em> meaning all GPUs will be used from this node.</li><li>Then, the total_gpus parameter indicates the total number of GPUs in the entire DDP training. I am assuming one process per GPU, so this will be the total number of processes involved in this DDP training. For example, in a distributed training of 2 nodes with 3 GPUs each, the total_gpus would be 3+3=6. We will use this value later.</li><li>The nr parameter sets up the ranking ids for the GPUs in this node. If we are the MASTER node, we will always have the rank 0. We must provide these values because each node is not able to know how many processes (or GPUs) will take part in the whole training beforehand. For instance, if we are the MASTER and we are using 2 GPUs in this node, we should set up nr 0,1. In case we are launching this script in a secondary node (not the MASTER) with the same number of GPUs, we could set up as nr 1,2.</li><li>After that, the ethernet_adaptersets up the network adapter you are going to use in this node to communicate with the other nodes. In Ubuntu (our operating system), you can type sudo lshw -class network -short to list all the ethernet adapters in your computer. The <em>Device</em> column is the value we need in this case. For example, ethernet_adapter enp2s0</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*G140nVnYUSEtXsWYrE189Q.png" /></figure><ul><li>The master_IP and the master_PORT are quite trivial. The MASTER takes control of all the training. You should set these parameters when launched in a distributed manner.</li><li>script.py is the file that will be launched in a distributed manner and it should contain your training code.</li></ul><p>Apart from those changes, I also modified the WORLD_SIZE environment variable to be aligned with the total_gpus parameters we saw before. In fastai, this value is set, at most, to the number of GPUs in one node. Besides this, the MASTER_ADDR and the MASTER_PORT environment variables are set up according to the values provided as parameters:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6cac3b168ead456d0ce33b69a82d662/href">https://medium.com/media/f6cac3b168ead456d0ce33b69a82d662/href</a></iframe><p>I added a condition to check if the <strong>number of GPUs to use in this node</strong> is equal to the <strong>number of ranking ids</strong> in the same node. Otherwise, we exit as they must match:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5eefe91766e8890652c12bf260efb10b/href">https://medium.com/media/5eefe91766e8890652c12bf260efb10b/href</a></iframe><p>Lastly, I also modified the loop to adequate it to the new configuration as well as passing it to the <em>script.py </em>two more parameters: the cuda_idand the ethernet_adapter:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae945ef093104c37aaa828438bb1cd94/href">https://medium.com/media/ae945ef093104c37aaa828438bb1cd94/href</a></iframe><p>Okay, good job for reaching this point! <strong>That was a tough part but you must understand all those concepts before taking your own DDP training.</strong></p><p>Now let’s move to the <em>script.py </em>that will contain all the powerful code we want to run.</p><h3>Distrib.py file</h3><p>In the fastai GitHub repository, there is a distributed training script example that we are going to slightly modify to make it run across all the nodes. You can find the original one <a href="https://github.com/fastai/fastai/blob/master/nbs/examples/distrib.py">here</a>.</p><p>Our file will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3305613f158eaf474bbf42510bf91c8f/href">https://medium.com/media/3305613f158eaf474bbf42510bf91c8f/href</a></iframe><p>Notice that, first, I changed the rank0_first function. I added the cuda_id parameter, which is then passed to the distrib_ctx function. This way I set up thecuda_idcorrectly for the specific process launching this file. Remember, one process takes control of one GPU.</p><p>The rest of the function remains the same as the<em> distrib.py</em> file of fastai:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b1fc5e567f9c0b1748bed3c6ec4317c/href">https://medium.com/media/8b1fc5e567f9c0b1748bed3c6ec4317c/href</a></iframe><p>Inside the main function, the NCCL_SOCKET_IFNAME environment variable is set up through the value provided as parameter. Then, I pass the cuda_id value to the distrib_ctx function. That is the change I have just mentioned in the rank0_first function and it is applied in the last line of code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/713a3342dcc88e167f488114498ddc19/href">https://medium.com/media/713a3342dcc88e167f488114498ddc19/href</a></iframe><p>Be aware that the python code now is in a <em>main </em>function as it is called by the <em>launch.py</em> file we saw before.</p><h3>Final comments and example of use</h3><p><strong>At this point, you should be able to launch your distributed training in fastai successfully.</strong></p><p>Before the final demo and usage example, I want to highlight some <strong>points </strong>to be aware of:</p><ul><li>Each <em>launch.py</em> and <em>distrib.py</em> (or the name of your script) must be <strong>equal</strong> in <strong>every node</strong> taking part in the DDP training.</li><li>I recommend <strong>starting up</strong> the DDP training in the <strong>MASTER </strong>node to follow a normal workflow. However, you can launch the files in any order.</li><li>You should set up all the <strong>parameters correctly</strong> in each node. Otherwise, there will be an error or the DDP training will not be launched.</li><li>In <strong>fastai</strong>, the <strong>output </strong>in the terminal corresponds to the <strong>MASTER</strong> node. Remember that in a DDP training, each GPU has its own data (for training and validation) and the metric displayed belongs to the rank 0 (the MASTER).</li><li>In relation to this last point, if you want to make specific tasks with one GPU, you can introduce a condition checking the rank id. For example, if (rank_distrib()==0):</li><li>A <strong>great trick</strong> found in <a href="https://leimao.github.io/blog/PyTorch-Distributed-Training/">this post</a> already mentioned is to be able to <strong>kill all the processes</strong> involved in the DDP training. This is useful when something is wrong or missing and we should restart the DDP training. You should clear things up before starting again. To do so, use the following command where script is the name of the distributed file (in this post <em>distrib.py</em>): kill $(ps aux | grep script | grep -v grep | awk &#39;{print $2}&#39;)</li></ul><p>Finally, I am going to show an <strong>example </strong>of launching a DDP training in fastai. I will assume that I have 3 nodes with 2 GPUs each and I want to use all of them as part of the DDP training. Here is a little scheme to make things easier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fH9HhHMvwZbxq6fDccyzQ.jpeg" /><figcaption>The scheme for our final example</figcaption></figure><p>In the <strong>first node</strong>, I will launch the following command. I set this node as the <strong>MASTER </strong>node. The <em>launch.py</em> and the <em>distrib.py</em> are in the same folder and I launch the commands from that folder.</p><p>python -m launch --gpus_to_use all --total_gpus 6 --nr 0,1 --ethernet_adapter enp65s0f1 --master_IP 10.92.130.54 --master_PORT 10000 distrib.py</p><p>In the <strong>second node</strong>, I will launch the following one:</p><p>python -m launch --gpus_to_use all --total_gpus 6 --nr 2,3 --ethernet_adapter enp3s0 --master_IP 10.92.130.54 --master_PORT 10000 distrib.py</p><p>And in the <strong>third one</strong>, the following one:</p><p>python -m launch --gpus_to_use all --total_gpus 6 --nr 4,5 --ethernet_adapter enp65s63 --master_IP 10.92.130.54 --master_PORT 10000 distrib.py</p><p>After some seconds of establishing connections and initialization you should visualize something like this in the <strong>MASTER node terminal</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*G25tZ8HhjkWVc_yz8C5N9g.png" /></figure><p>This means that everything is going well. You can check this out with the nvidia-smi command as well and ensuring the GPUs are working hard enough!</p><p>After the training is completed, you will see information about the training phase as usual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*Tql47ZC3ZyG2ESrQ6Z5YtA.png" /></figure><p>Remember that these <strong>values </strong>will only be displayed in the <strong>MASTER </strong>node terminal and they correspond to the <strong>loss and metrics of the rank 0</strong>. It should extrapolate to a general behaviour but I recommend saving the model and evaluating it with the whole validation set to get the final metric.</p><p>I appreciate you have reached this point and hopefully you managed to launch your DDP training successfully.</p><p>If you have any comments, questions or suggestions do not hesitate to leave any kind of feedback below. I am sure there is room for improvement!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=299c0ef56e2f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/naia-blog/distributed-training-across-multiple-nodes-in-fastai-299c0ef56e2f">Distributed Training across multiple nodes in fastai</a> was originally published in <a href="https://medium.com/naia-blog">NaIA: Machine Learning at the heart of Public Healthcare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>