<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ganesh on Medium]]></title>
        <description><![CDATA[Stories by Ganesh on Medium]]></description>
        <link>https://medium.com/@dsnikki07?source=rss-0e0bab1bbe9d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*g95rVvNXgaVvcZnvWeMslA.jpeg</url>
            <title>Stories by Ganesh on Medium</title>
            <link>https://medium.com/@dsnikki07?source=rss-0e0bab1bbe9d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:32:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dsnikki07/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 7…]]></title>
            <link>https://medium.com/@dsnikki07/building-a-high-performance-computing-hpc-cluster-from-scratch-a-step-by-step-guide-part-8-06a2b150188f?source=rss-0e0bab1bbe9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/06a2b150188f</guid>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[slums]]></category>
            <category><![CDATA[hpc]]></category>
            <dc:creator><![CDATA[Ganesh]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 05:54:55 GMT</pubDate>
            <atom:updated>2026-08-24T14:58:18.694Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 7: Advanced Slurm Configuration — Accounting, QoS, Fairshare, and Priority Management</h3><p>In the previous parts of this series, we established a solid foundation for the cluster: operating system installation, authentication, shared storage, basic Slurm job scheduling, and environment modules with LMOD.</p><p>While basic Slurm allows users to submit jobs and allocate resources, a production-grade HPC cluster serving multiple users and research groups requires more sophisticated management. This part focuses on <strong>advanced Slurm features</strong> that bring professional-level control, fairness, accountability, and flexibility to your 12-node cluster.</p><h3>What You Will Learn in This Part</h3><ul><li>How to set up the <strong>Slurm Accounting Database</strong> (slurmdbd) to track resource usage over time.</li><li>Managing users, accounts, and associations using sacctmgr.</li><li>Implementing <strong>Quality of Service (QoS)</strong> to create different job priority levels and limits.</li><li>Configuring <strong>Fairshare</strong> policies so that resource allocation feels equitable across users and groups.</li><li>Understanding how Slurm calculates job <strong>priority</strong> using its multi-factor formula.</li><li>Practical commands, expected outputs, and real-world best practices.</li></ul><p>These features allow cluster administrators to enforce usage policies, generate usage reports, prioritize critical jobs, and prevent any single user from monopolizing the cluster.</p><h3>1. Setting Up Slurm Accounting Database (slurmdbd)</h3><p>The accounting database records every job’s resource consumption (CPU time, GPU hours, memory, walltime, etc.). This data powers usage reports, fairshare calculations, and billing-like systems.</p><h4>Step 1: Install and Secure MariaDB on the HeadNode</h4><pre>sudo dnf install mariadb-server mariadb -y<br>sudo systemctl enable --now mariadb</pre><p>This installs the MariaDB database server and starts it. Then secure it:</p><pre>sudo mysql_secure_installation</pre><p>Follow the prompts to set a strong root password and remove test databases.</p><h4>Step 2: Create the Slurm Database and Dedicated User</h4><pre>sudo mysql -u root -p</pre><p>Inside the MySQL prompt, run:</p><pre>CREATE DATABASE slurm_acct_db;<br>CREATE USER &#39;slurm&#39;@&#39;localhost&#39; IDENTIFIED BY &#39;YourStrongDatabasePassword123!&#39;;<br>GRANT ALL PRIVILEGES ON slurm_acct_db.* TO &#39;slurm&#39;@&#39;localhost&#39; WITH GRANT OPTION;<br>FLUSH PRIVILEGES;<br>EXIT;</pre><p>This creates a dedicated database and a restricted user for Slurm. Using a separate user improves security.</p><h4>Step 3: Configure slurmdbd</h4><pre>sudo nano /etc/slurm/slurmdbd.conf</pre><p>Add the following content:</p><pre>ArchiveEvents=yes<br>ArchiveJobs=yes<br>ArchiveResvs=yes<br>AuthType=auth/munge<br>DbdHost=headnode<br>DbdPort=6819<br>SlurmUser=slurm<br>StorageType=accounting_storage/mysql<br>StorageHost=localhost<br>StorageUser=slurm<br>StoragePass=YourStrongDatabasePassword123!<br>StorageLoc=slurm_acct_db</pre><ul><li>AuthType=auth/munge: Uses the MUNGE key (already configured) for secure communication.</li><li>StorageType=accounting_storage/mysql: Tells Slurm to store data in MariaDB.</li></ul><p>Start the service:</p><pre>sudo systemctl enable --now slurmdbd<br>sudo systemctl status slurmdbd</pre><p>You should see the service running without errors.</p><h4>Step 4: Update Main Slurm Configuration</h4><p>Edit /etc/slurm/slurm.conf on the HeadNode and add/replace these lines:</p><pre>AccountingStorageType=accounting_storage/slurmdbd<br>AccountingStorageHost=headnode<br>AccountingStorageUser=slurm<br>JobAcctGatherType=jobacct_gather/cgroup<br>JobAcctGatherFrequency=30</pre><p>Restart the controller:</p><pre>sudo systemctl restart slurmctld</pre><h3>2. User and Account Management with sacctmgr</h3><p>sacctmgr is the administrative command for managing users, accounts (groups), and their relationships.</p><h4>Common Administrative Commands</h4><p><strong>Create a new account (group):</strong></p><pre>sudo sacctmgr add account name=groupa desc=&quot;Research Group A&quot; Organization=Lab</pre><p>This creates a logical container for users.</p><p><strong>Add users to an account:</strong></p><pre>sudo sacctmgr add user name=user1 account=groupa adminlevel=Administrator<br>sudo sacctmgr add user name=user2 account=groupa</pre><p><strong>View current associations:</strong></p><pre>sacctmgr show assoc format=cluster,account,user,partition,qos<br>sacctmgr show user</pre><p>Expected output will show a table listing users, their accounts, and assigned partitions/QoS.</p><p><strong>Set resource limits on a user:</strong></p><pre>sudo sacctmgr modify user where name=user1 set MaxSubmitJobs=50 MaxJobs=10 GrpTRES=cpu=200</pre><h3>3. Quality of Service (QoS)</h3><p>QoS lets you define job classes with different priorities, maximum runtimes, and preemption rules.</p><h4>Create QoS Levels</h4><pre># High priority QoS for urgent jobs<br>sudo sacctmgr add qos name=urgent priority=10000 MaxWall=4:00:00 MaxTRES=gpu=8</pre><pre># Standard/default QoS<br>sudo sacctmgr add qos name=normal priority=1000 MaxWall=72:00:00 GrpTRES=cpu=400</pre><pre># Low priority for long-running jobs<br>sudo sacctmgr add qos name=low priority=100 MaxWall=14-00:00:00 Preempt=urgent,normal</pre><p><strong>Assign QoS to accounts or users:</strong></p><pre>sudo sacctmgr modify account groupa set qos=normal,low defaultqos=normal<br>sudo sacctmgr modify user user1 set qos+=urgent</pre><p>Users can specify QoS in their job scripts:</p><pre>#SBATCH --qos=urgent</pre><h3>4. Fairshare Configuration</h3><p>Fairshare ensures users and groups who have used fewer resources recently receive higher scheduling priority.</p><p>Add the following to slurm.conf:</p><pre>PriorityType=priority/multifactor<br>PriorityWeightFairshare=100000<br>PriorityWeightAge=2000<br>PriorityWeightQOS=1000<br>PriorityWeightPartition=1000<br>PriorityDecayHalfLife=14-0</pre><p><strong>Assign fairshare shares to accounts:</strong></p><pre>sudo sacctmgr modify account groupa set Fairshare=100<br>sudo sacctmgr modify account groupb set Fairshare=50</pre><p><strong>Check fairshare status:</strong></p><pre>sshare<br>sshare -l -a</pre><p>The output shows normalized usage and effective priority shares for each user/account. Lower usage = higher fairshare factor.</p><h3>5. Understanding Slurm’s Job Priority Formula</h3><p>Slurm calculates priority using a weighted multi-factor formula:</p><p><strong>Priority = (AgeFactor × WeightAge) + (FairshareFactor × WeightFairshare) + (QOSFactor × WeightQOS) + (PartitionFactor × WeightPartition) + …</strong></p><ul><li><strong>AgeFactor</strong>: Increases as a job waits longer in the queue.</li><li><strong>FairshareFactor</strong>: Based on recent usage. Formula roughly: 2^(-usage / share). Users who have used less get boosted.</li><li><strong>QOSFactor / PartitionFactor</strong>: Static boosts from QoS and partition priority.</li></ul><p><strong>View current priorities:</strong></p><pre>sprio                  # Summary<br>sprio -l -S -1         # Detailed, sorted by priority<br>scontrol show job &lt;jobid&gt;</pre><h3>Best Practices and Final Recommendations</h3><ul><li>Start with conservative limits and gradually relax them based on real usage.</li><li>Regularly review reports using:</li></ul><pre>sreport cluster utilization sacct -u user1 --starttoday</pre><ul><li>Document your cluster’s QoS and fairshare policy for all users.</li><li>Back up the accounting database regularly.</li><li>Test changes with non-critical jobs before applying to production.</li></ul><p>These advanced features make your cluster scalable, fair, and auditable — ready for multiple research teams.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06a2b150188f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 6…]]></title>
            <link>https://medium.com/@dsnikki07/building-a-high-performance-computing-hpc-cluster-from-scratch-a-step-by-step-guide-part-6-edc8fa2b1202?source=rss-0e0bab1bbe9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/edc8fa2b1202</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[slurm]]></category>
            <category><![CDATA[hpc]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[slurm-job]]></category>
            <dc:creator><![CDATA[Ganesh]]></dc:creator>
            <pubDate>Sat, 13 Jun 2026 23:09:00 GMT</pubDate>
            <atom:updated>2026-06-13T23:09:00.300Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 6: Slurm Workload Manager Setup</h3><p>In Part 5, we configured LMOD that allows users to easily load different versions of software without conflicts across the cluster.</p><p>In this part, we install and configure <strong>Slurm, </strong>the most widely used open-source workload manager in HPC. Slurm handles job scheduling, resource allocation (CPUs, GPUs, memory), queuing, and fair-share policies across all 12 compute nodes.</p><h3>Why Slurm?</h3><p>Slurm (Simple Linux Utility for Resource Management) is the de-facto standard in research and academic HPC. It excels at:</p><ul><li>Allocating CPUs and GPUs efficiently</li><li>Supporting partitions (our High-Density, Balanced GPU, and CPU-only tiers)</li><li>Fair-share scheduling and Quality of Service (QoS)</li><li>Integration with MUNGE (already set up in Part 3)</li></ul><h3>Step 1: Create the Slurm System User (All Nodes)</h3><p>This user must exist with the <strong>same UID/GID</strong> on every node.</p><pre>sudo groupadd -g 100 slurm<br>sudo useradd -m -c &quot;Slurm Workload Manager&quot; -d /var/lib/slurm -u 100 -g slurm -s /bin/false slurm</pre><ul><li>groupadd -g 100: Creates a group with a specific Group ID (any number below 1000 is a common convention for system services).</li><li>useradd: Creates the slurm user with home directory /var/lib/slurm, no login shell, and matching UID.</li></ul><p>Verify:</p><pre>id slurm</pre><h3>Step 2: Slurm Installation and Configuration on the HeadNode</h3><h4>1. Install Slurm Controller</h4><pre>sudo dnf install slurm-slurmctld -y</pre><h4>2. Create Required Directories</h4><pre>sudo mkdir -p /etc/slurm /var/spool/slurmctld /var/log/slurm<br>sudo chown slurm:slurm /var/spool/slurmctld /var/log/slurm</pre><h4>3. Create Main Configuration File — /etc/slurm/slurm.conf</h4><pre>sudo nano /etc/slurm/slurm.conf</pre><p>Note:</p><p>To check on the hardware config on compute nodes, just run the following command and copy the result into slurm.conf file on Headnode</p><pre>sudo slurmd -C</pre><p>Here is a solid base configuration (customize node names and resources as needed):</p><pre>ClusterName=researchcluster<br>SlurmctldHost=headnode<br>GresTypes=gpu<br><br>ProctrackType=proctrack/cgroup<br>TaskPlugin=task/cgroup<br>SelectType=select/cons_res<br>SelectTypeParameters=CR_Core_Memory<br><br>SchedulerType=sched/backfill<br># SchedulerType=sched/builtin   # Use for simple testing<br><br>SlurmctldPidFile=/var/run/slurmctld.pid<br>SlurmdPidFile=/var/run/slurmd.pid<br>SlurmdSpoolDir=/var/spool/slurmd<br>StateSaveLocation=/var/spool/slurmctld<br>TmpFS=/stash<br><br># Logging<br>SlurmctldDebug=info<br>SlurmctldLogFile=/var/log/slurm/slurmctld.log<br>SlurmdDebug=info<br>SlurmdLogFile=/var/log/slurm/slurmd.log<br><br># Node Definitions (examples - adapt to your actual hostnames)<br>NodeName=headnode CPUs=64 RealMemory=500000 State=UNKNOWN<br><br># High-Density GPU Nodes (4 GPUs each)<br>NodeName=high-density-[01-02] CPUs=112 RealMemory=1030000 Gres=gpu:nvidia:4 State=UNKNOWN<br><br># Balanced GPU Nodes (1 GPU each)<br>NodeName=gpu-standard-[01-05] CPUs=112 RealMemory=1030000 Gres=gpu:nvidia:1 State=UNKNOWN<br><br># CPU-only Nodes<br>NodeName=cpu-only-[01-05] CPUs=64 RealMemory=786000 State=UNKNOWN<br><br># Partitions (Tiers)<br>PartitionName=high-density Nodes=high-density-[01-02] Default=NO MaxTime=7-00:00:00 State=UP PriorityTier=20<br>PartitionName=gpu-standard Nodes=gpu-standard-[01-05] Default=YES MaxTime=INFINITE State=UP PriorityTier=10<br>PartitionName=cpu-only Nodes=cpu-only-[01-05] Default=NO MaxTime=14-00:00:00 State=UP PriorityTier=5</pre><p><strong>Key directives explained</strong>:</p><ul><li>GresTypes=gpu: Enables Generic Resource Scheduling for GPUs.</li><li>SelectTypeParameters=CR_Core_Memory: Allows core + memory based allocation.</li><li>PartitionName: Defines our three tiers with different priorities and time limits.</li></ul><h4>4. Create cgroup Configuration</h4><pre>sudo nano /etc/slurm/cgroup.conf</pre><pre>CgroupAutomount=yes<br>ConstrainCores=yes<br>ConstrainRAMSpace=yes<br>ConstrainDevices=yes<br>AllowedRAMSpace=90 #adjust to your needs</pre><h3>Step 3: GPU Configuration (GRES) on Compute Nodes</h3><p>On <strong>every GPU node</strong>, create the GRES file:</p><pre>sudo nano /etc/slurm/gres.conf</pre><p><strong>For High-Density nodes (4 GPUs):</strong></p><pre>Name=gpu Type=nvidia File=/dev/nvidia[0-3]</pre><p><strong>For Balanced GPU nodes (1 GPU):</strong></p><pre>Name=gpu Type=nvidia File=/dev/nvidia0</pre><h3>Step 4: Firewall Configuration (All Nodes)</h3><p><strong>On HeadNode:</strong></p><pre>sudo firewall-cmd --permanent --add-port=6817/tcp<br>sudo firewall-cmd --reload</pre><p><strong>On Compute Nodes:</strong></p><pre>sudo firewall-cmd --permanent --add-port=6818/tcp<br>sudo firewall-cmd --reload</pre><h3>Step 5: Copy Configuration Files to All Compute Nodes</h3><p>From the HeadNode, copy the files to every compute node (use a loop or Ansible for efficiency):</p><pre>for node in gpu-standard-01 gpu-standard-02 ... cpu-only-01 ...; do<br>    scp /etc/slurm/slurm.conf $node:/etc/slurm/<br>    scp /etc/slurm/cgroup.conf $node:/etc/slurm/<br>    scp /etc/slurm/gres.conf $node:/etc/slurm/   # Only for GPU nodes<br>done</pre><h3>Step 6: Start Slurm Services</h3><p><strong>On HeadNode:</strong></p><pre>sudo systemctl enable --now slurmctld<br>sudo systemctl status slurmctld</pre><p><strong>On All Compute Nodes:</strong></p><pre>sudo dnf install slurm-slurmd -y<br>sudo systemctl enable --now slurmd<br>sudo systemctl status slurmd</pre><h3>Step 7: Verification Commands</h3><p>From the HeadNode, run these important checks:</p><pre>sinfo                    # View all nodes and their state<br>sinfo -N -l              # More detailed node info<br>scontrol show nodes      # Detailed node configuration<br>scontrol show partition  # View partitions</pre><p>A healthy output should show all nodes as idle or mixed.</p><p>Test submitting a simple job:</p><pre>srun --partition=gpu-standard --gres=gpu:1 --nodelist=gpu-standard-01 hostname</pre><h3>Best Practices</h3><ul><li>Keep slurm.conf and cgroup.conf identical across all nodes.</li><li>Use consistent node naming conventions (e.g., high-density-01).</li><li>Back up /etc/slurm/ regularly.</li><li>For more control, enable Slurm accounting (slurmdbd) and set up Quality of Service (QoS) rules.</li><li>Monitor logs: tail -f /var/log/slurm/slurmctld.log</li></ul><h3>What’s Next?</h3><p>In <strong>Part 6</strong>, we will continue with more Slurm setup.</p><p>You now have a working job scheduler! Users can submit jobs across your 12-node cluster with proper GPU awareness.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edc8fa2b1202" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 5…]]></title>
            <link>https://medium.com/@dsnikki07/building-a-high-performance-computing-hpc-cluster-from-scratch-a-step-by-step-guide-part-5-6bb49a9f5b5a?source=rss-0e0bab1bbe9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bb49a9f5b5a</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[hpc-data-center]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[hpc-cluster]]></category>
            <category><![CDATA[hpc]]></category>
            <dc:creator><![CDATA[Ganesh]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 03:57:57 GMT</pubDate>
            <atom:updated>2026-06-10T03:57:57.407Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 5: LMOD Environment Modules System</h3><p>In Part 4, we configured NFS shared storage so that all nodes can access common directories.</p><p>In this part, we set up <strong>LMOD</strong> — the modern environment modules system. LMOD is one of the most critical tools for usability in any professional HPC cluster.</p><h3>Why Do We Need LMOD?</h3><p>Modern HPC clusters host dozens (or hundreds) of software packages, libraries, compilers, and toolchains that often have multiple versions of the same software (e.g., CUDA 12.4, 12.6; Python 3.10, 3.11, 3.12; TensorFlow, PyTorch, etc.).</p><p>Without a modules system:</p><ul><li>Users would manually modify PATH, LD_LIBRARY_PATH, PYTHONPATH, etc., leading to conflicts and “it works on my machine” problems.</li><li>Reproducibility across nodes and over time becomes nearly impossible.</li></ul><p><strong>LMOD</strong> (Lua-based Modular Environment) solves this by allowing users to dynamically load and unload software environments with simple commands like:</p><ul><li>module load cuda/12.6</li><li>module load python/3.11</li><li>module avail</li><li>module list</li></ul><p>LMOD is developed by TACC (Texas Advanced Computing Center) and has become the de-facto standard in most research HPC centers due to its speed, hierarchical module support, and excellent dependency handling.</p><h3>Part 6.1: LMOD Installation on the HeadNode</h3><h4>1. Install Build Dependencies</h4><pre>sudo dnf install -y lua lua-devel lua-posix lua-filesystem tcl-devel</pre><ul><li>These packages provide the Lua runtime and development libraries that LMOD depends on.</li></ul><h4>2. Download and Build LMOD</h4><p>We install LMOD into /opt/apps following best practices (this allows easy future upgrades):</p><pre>cd /tmp<br>wget https://github.com/TACC/Lmod/archive/refs/tags/8.7.tar.gz<br>tar -xvf 8.7.tar.gz<br>cd Lmod-8.7</pre><pre>./configure --prefix=/opt/apps<br>make -j$(nproc)<br>sudo make install</pre><ul><li>— prefix=/opt/apps: Installs LMOD into a versioned directory structure.</li><li>make -j$(nproc): Uses all available CPU cores to speed up compilation.</li></ul><h4>3. Create Symbolic Link for Easy Upgrades</h4><pre>sudo ln -s /opt/apps/lmod/lmod /opt/apps/lmod/lmod-current</pre><h4>4. Initialize LMOD System-Wide</h4><p>Create symbolic links so that every user automatically gets the module command:</p><pre>sudo ln -s /opt/apps/lmod/lmod/init/profile /etc/profile.d/z00_lmod.sh<br>sudo ln -s /opt/apps/lmod/lmod/init/cshrc /etc/profile.d/z00_lmod.csh</pre><h4>5. Create Module Directory Structure</h4><pre>sudo mkdir -p /opt/modulefiles/Core<br>sudo mkdir -p /etc/lmod</pre><h4>6. Configure Module Search Path</h4><pre>echo &quot;/opt/modulefiles/Core&quot; | sudo tee /etc/lmod/.modulespath</pre><h4>7. Test on HeadNode</h4><pre>source /etc/profile.d/z00_lmod.sh<br>module avail<br>module --version</pre><h3>Part 6.2: LMOD Setup on All Compute Nodes</h3><p>Since /opt/apps and /opt/modulefiles will be shared via NFS (from Part 4), we only need to configure the shell initialization on each compute node.</p><h4>1. Install Runtime Dependencies</h4><pre>sudo dnf install -y lua-filesystem lua-posix</pre><h4>2. Link Initialization Scripts</h4><pre>sudo ln -s /opt/apps/lmod/lmod/init/profile /etc/profile.d/z00_lmod.sh</pre><h4>3. Refresh Environment and Test</h4><pre>source /etc/profile.d/z00_lmod.sh<br>export MODULEPATH=/opt/modulefiles/Core<br>module avail</pre><p>Repeat this process (or use Ansible) on all 12 compute nodes.</p><h3>Part 6.3: Creating Your First Modulefiles</h3><p>Modulefiles are written in Lua and tell the shell how to modify environment variables when a module is loaded.</p><h4>Example: Create a CUDA Module</h4><ol><li>Create directory structure:</li></ol><pre>sudo mkdir -p /opt/modulefiles/Core/cuda</pre><ol><li>Create the module file:</li></ol><pre>sudo nano /opt/modulefiles/Core/cuda/12.6.lua</pre><pre>help([[ CUDA Toolkit 12.6]])<br>whatis(&quot;Version: 12.6&quot;)<br>whatis(&quot;Description: NVIDIA CUDA Toolkit for GPU computing&quot;)<br>local root = &quot;/usr/local/cuda-12.6&quot;<br>prepend_path(&quot;PATH&quot;, root .. &quot;/bin&quot;)<br>prepend_path(&quot;LD_LIBRARY_PATH&quot;, root .. &quot;/lib64&quot;)<br>prepend_path(&quot;LIBRARY_PATH&quot;, root .. &quot;/lib64&quot;)<br>prepend_path(&quot;INCLUDE&quot;, root .. &quot;/include&quot;)<br>prepend_path(&quot;MANPATH&quot;, root .. &quot;/doc/man&quot;)<br>setenv(&quot;CUDA_HOME&quot;, root)<br>setenv(&quot;CUDA_VERSION&quot;, &quot;12.6&quot;)</pre><p><strong>Key Lua functions explained</strong>:</p><ul><li>prepend_path(): Adds directories to environment variables without overwriting existing ones.</li><li>setenv(): Sets specific environment variables.</li><li>help() and whatis(): Provide information when users run module help or module whatis.</li></ul><h4>Make the Module Available</h4><pre>module avail<br>module load cuda/12.6<br>nvcc --version<br>module unload cuda/12.6</pre><h3>Part 6.4: Advanced LMOD Features &amp; Best Practices</h3><ul><li><strong>Hierarchical Modules</strong>: You can create module categories (e.g., Core, Compiler, MPI, Applications).</li><li><strong>Spider Cache</strong> (for performance): When you have many modules, enable spider cache:</li></ul><pre>sudo mkdir -p /opt/modulefiles/cache module spider</pre><ul><li><strong>Default Versions</strong>: Create a .version file inside a module directory to set a default version.</li><li><strong>Module Collections</strong>: Users can save their favorite combinations with module save myenv and restore with module restore myenv.</li><li><strong>Naming Convention</strong>: Use package/version (e.g., gcc/13.2, openmpi/4.1.6).</li></ul><p><strong>Recommended Practices</strong>:</p><ul><li>Keep all software installations in /opt/apps/&lt;package&gt;/&lt;version&gt;.</li><li>Document every module with meaningful help() text.</li><li>Use versioned modulefiles (never overwrite old versions).</li><li>Regularly run module spider to update the cache.</li><li>Back up the entire /opt/modulefiles directory.</li></ul><h4>Verification Across the Cluster</h4><p>From any node (including compute nodes):</p><pre>module avail<br>module load cuda/12.6<br>echo $PATH | grep cuda<br>module list<br>module unload cuda</pre><h3>Part 6.5: How to use LMOD?</h3><h4>1. Core Module Commands Reference</h4><p>Every interaction with the software stack uses the module command (or the shorthand ml).</p><p><strong>Finding Software</strong></p><ul><li><strong>module avail (or ml av)</strong>: Lists all software packages currently visible and available. <em>Tip</em>: Filter with a keyword: module avail cuda</li><li><strong>module spider</strong>: The most powerful search tool. It searches <strong>all</strong> modules, even those hidden behind dependencies. Example: module spider python</li></ul><p><strong>Managing Your Environment</strong></p><ul><li><strong>module load &lt;package&gt; (or ml &lt;package&gt;)</strong>: Loads the software into your current shell session. Example: module load git/2.47.3</li><li><strong>module list (or ml):</strong> Shows currently loaded modules in your session.</li><li><strong>module unload &lt;package&gt;:</strong> Removes a module from your environment.</li><li><strong>module purge:</strong> Unloads <strong>all</strong> modules — useful for starting clean.</li><li><strong>module swap:</strong> old/new Replaces one version with another seamlessly.</li></ul><p><strong>Handy Shorthand (ml)</strong>:</p><ul><li>ml → module list</li><li>ml git → module load git</li><li>ml -git → module unload git</li><li>ml av → module avail</li></ul><h4>2. Understanding Module Output</h4><p>When you run module avail, you will see output similar to this:</p><pre>---------------------------- /opt/modulefiles/Core -----------------------------<br>   <br>   gcc/24.1         python/3.7(L)       testutils/1.0<br><br>--------------------- /opt/apps/lmod/modulefiles/Core ---------------------<br>   lmod    settarg<br><br>  Where:<br>   L:  Module is loaded<br>   D:  Default module<br><br>If the avail list is too long consider trying:<br><br>&quot;module --default avail&quot; or &quot;ml -d av&quot; to just list the default modules.<br>&quot;module overview&quot; or &quot;ml ov&quot; to display the number of modules for each name.<br><br>Use &quot;module spider&quot; to find all possible modules and extensions.<br>Use &quot;module keyword key1 key2 ...&quot; to search for all possible modules matching<br>any of the &quot;keys&quot;.</pre><ul><li><strong>(L)</strong> = Currently loaded in your session</li><li><strong>(D)</strong> = Default version (loaded if you don’t specify a version)</li><li>/opt/modulefiles/Core = Primary location for your software stack</li></ul><p><strong>Useful Commands</strong>:</p><pre>module --default avail     # Show only default modules<br>module overview            # Summary of available modules<br>module keyword cuda        # Search by keyword</pre><h3>Part 6.6: Best Practices for Using Modules</h3><p><strong>Rule 1: Always Load Modules in Job Scripts</strong></p><p>Instead of putting absolute paths like /opt/apps/python/3.7/bin/python into your job submission or execution scripts, always load the module at the beginning of your workflow:</p><pre># Clean environment<br>module purge</pre><pre># Load exact versions needed<br>module load gcc/24.1<br>module load pyhton/3.7</pre><pre># Your commands<br>python train_model.py</pre><p>This ensures perfect reproducibility even months later.</p><p><strong>Rule 2: Use module purge Before Loading New Toolchains</strong></p><p>This prevents version conflicts between Python, CUDA, compilers, etc.</p><p><strong>Rule 3: Create Module Collections</strong> (Advanced)</p><pre>module save my_ml_env          # Save current loaded modules<br>module restore my_ml_env       # Restore later</pre><h3>Part 6.6: Troubleshooting Common Issues</h3><ol><li><strong>“Command not found”</strong> after loading a module → Run hash -r to refresh Bash’s command cache.</li><li><strong>Module doesn’t appear in module avail</strong> → Use module spider &lt;name&gt; — it may require a prerequisite (e.g., load a compiler first).</li><li><strong>Library or version conflicts</strong> → Run module purge and reload only the minimum required modules.</li><li><strong>Module list is too long</strong> → Use module spider or module keyword for targeted searches.</li></ol><h3>What’s Next?</h3><p>In <strong>Part 6</strong>, we will install and configure Slurm, the powerful workload manager that will let users submit jobs across all 12 compute nodes with proper CPU and GPU resource allocation.</p><p>Your cluster now has a professional, user-friendly software environment thanks to LMOD!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bb49a9f5b5a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 4…]]></title>
            <link>https://medium.com/@dsnikki07/building-a-high-performance-computing-hpc-cluster-from-scratch-a-step-by-step-guide-part-4-f0cda80505b7?source=rss-0e0bab1bbe9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0cda80505b7</guid>
            <category><![CDATA[hpc]]></category>
            <category><![CDATA[nfs-server]]></category>
            <category><![CDATA[network-file-system]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Ganesh]]></dc:creator>
            <pubDate>Sat, 16 May 2026 22:57:26 GMT</pubDate>
            <atom:updated>2026-05-17T00:01:07.130Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 4: NFS Shared Storage Configuration</h3><p>In Part 3, we configured MUNGE for secure inter-node authentication and LDAP + SSSD for centralized user management. In this part, we configure <strong>NFS (Network File System) </strong>— the shared storage layer that allows users to access the same files (home directories, projects, and data) from any node in the cluster.</p><p>This is a critical step because it enables users to seamlessly access their files from any node in the cluster (HeadNode or any of the 12 compute nodes).</p><h3>Why NFS in This Phase?</h3><p>NFSv4 is simple, reliable, and easy to deploy, making it an excellent <strong>Phase 1</strong> shared filesystem for most research clusters. Later, as your data grows beyond 50–100 TB, you can migrate to a parallel filesystem such as BeeGFS, Lustre, or Weka.</p><p>We will:</p><ul><li>Export shares from the <strong>HeadNode</strong></li><li>Mount them on all compute nodes using both static mounts (/etc/fstab) and <strong>autofs</strong> for better reliability</li></ul><h3>Planned Shared Directories</h3><p>We will create and share the following common directories:</p><ul><li>/scratch → Shared scratch space (temporary files, can be cleaned periodically)</li><li>/projects → Research project directories</li><li>/home → User home directories (via NFS)</li><li>/data → Large input/output datasets (read-heavy)</li><li>/software → Optional shared software area</li></ul><h3>Part 4.1: NFS Server Setup on the HeadNode</h3><h4>1. Install NFS Utilities</h4><pre>sudo dnf install nfs-utils -y</pre><ul><li>dnf install: Uses Red Hat’s package manager to download and install software.</li><li>nfs-utils: Contains all the tools and daemons required to run an NFS server.</li></ul><h4>2. Create the Shared Directories</h4><pre>sudo mkdir -p /scratch /projects /data /home<br>sudo chmod 755 /scratch /projects /data<br>sudo chmod 755 /home</pre><ul><li>mkdir -p: Creates directories and any necessary parent directories.</li><li>chmod 755: Sets permissions so the owner has full access and others can read/execute (standard for shared directories).</li></ul><h4>3. Configure NFS Exports</h4><p>Edit the exports file:</p><pre>sudo nano /etc/exports</pre><p>Add the following lines (adjust subnets to match your cluster network, e.g., 10.28.0.0/16):</p><pre>/scratch       10.28.0.0/16(rw,sync,no_root_squash,no_subtree_check)<br>/projects      10.28.0.0/16(rw,sync,no_root_squash,no_subtree_check)<br>/data          10.28.0.0/16(ro,sync,root_squash,no_subtree_check)<br>/home          10.28.0.0/16(rw,sync,no_root_squash,no_subtree_check)</pre><p><strong>Explanation of options</strong>:</p><ul><li>rw = Read and write access</li><li>ro = Read-only (for sensitive data)</li><li>sync = Write data to disk before replying (safer, slightly slower)</li><li>no_root_squash = Allows root on compute nodes to act as root on the share (required for Slurm)</li><li>no_subtree_check = Improves performance</li></ul><h4>4. Apply Exports and Start Services</h4><pre>sudo exportfs -rav</pre><ul><li>exportfs: Command to manage NFS exports.</li><li>-r: Re-export all directories (apply changes).</li><li>-a: Export all directories listed in /etc/exports.</li><li>-v: Verbose output (shows what was exported).</li></ul><pre>sudo systemctl enable --now nfs-server rpcbind</pre><ul><li>systemctl enable — now: Enables the service to start automatically at boot and starts it immediately.</li><li>nfs-server: Main NFS daemon.</li><li>rpcbind: Required supporting service for RPC (Remote Procedure Call) communication.</li></ul><h4>5. Configure Firewall on HeadNode</h4><pre>sudo firewall-cmd --permanent --add-service=nfs<br>sudo firewall-cmd --permanent --add-service=rpc-bind<br>sudo firewall-cmd --permanent --add-service=mountd<br>sudo firewall-cmd --reload</pre><ul><li>--permanent: Makes the rule survive reboots.</li><li>--add-service: Opens the necessary firewall ports for NFS.</li><li>--reload: Applies the new firewall rules immediately.</li></ul><h3>Part 4.2: NFS Client Setup on All Compute Nodes</h3><h4>1. Install NFS Utilities</h4><pre>sudo dnf install nfs-utils -y</pre><h4>2. Create Mount Points</h4><pre>sudo mkdir -p /scratch /projects /data /home</pre><h4>3. Mount NFS Shares Using /etc/fstab (Static Mount)</h4><p>Edit the fstab file:</p><pre>sudo nano /etc/fstab</pre><p>Add these lines at the end (replace headnode with your actual HeadNode hostname or IP):</p><pre>headnode:/scratch    /scratch    nfs    defaults,hard,rsize=32768,wsize=32768  0 0<br>headnode:/projects   /projects   nfs    defaults,hard,rsize=32768,wsize=32768  0 0<br>headnode:/data       /data       nfs    defaults,hard,ro,rsize=32768,wsize=32768 0 0<br>headnode:/home       /home       nfs    defaults,hard,rsize=32768,wsize=32768  0 0</pre><p><strong>Important options explained</strong>:</p><ul><li>defaults — Standard mount options</li><li>hard — If the server becomes unavailable, the client will keep retrying (recommended for HPC)</li><li>rsize / wsize — Read/write block size in bytes (32768 = 32KB improves performance)</li><li>timeo=600 — Timeout value in tenths of a second</li></ul><p>Mount all shares:</p><pre>sudo mount -a</pre><ul><li>mount -a: Mounts all filesystems listed in /etc/fstab that are not yet mounted.</li></ul><p>Verify:</p><pre>df -h | grep headnode</pre><h3>Part 4.3: Recommended — AutoFS for More Reliable Mounting</h3><p>Autofs automatically mounts shares when accessed and unmounts them after inactivity which is better for stability in large clusters.</p><h4>1. Install and Enable AutoFS</h4><pre>sudo dnf install autofs -y<br>sudo systemctl enable --now autofs</pre><h4>2. Create AutoFS Map File</h4><pre>sudo nano /etc/auto.share</pre><p>Add the following:</p><pre>/scratch     -rw,soft,intr,no_root_squash,rsize=32768,wsize=32768    headnode:/scratch<br>/projects    -rw,soft,intr,no_root_squash,rsize=32768,wsize=32768    headnode:/projects<br>/data        -ro,soft,intr,root_squash,rsize=32768,wsize=32768       headnode:/data<br>/home        -rw,soft,intr,no_root_squash,rsize=32768,wsize=32768    headnode:/home</pre><h4>3. Configure AutoFS Master Map</h4><pre>sudo nano /etc/auto.master</pre><p>Add at the bottom:</p><pre>/-    /etc/auto.share    --timeout=300<br></pre><ul><li>/- indicates a “direct map,” which allows you to mount directly to an absolute path.</li><li>— timeout=300 tells autofs to unmount the folder if it hasn’t been used for 300 seconds (prevents stale mount hangs).</li></ul><h4>3. Restart and Test AutoFS</h4><pre>sudo systemctl restart autofs</pre><p>Test by accessing the directories:</p><pre>ls /scratch<br>ls /projects</pre><h3>Verification Commands</h3><p>From any compute node:</p><pre>showmount -e headnode          # Shows what the server is exporting<br>df -h | grep headnode          # Shows mounted NFS shares<br>mount | grep nfs               # Detailed mount information</pre><h3>Best Practices and Troubleshooting</h3><ul><li>Use showmount -e headnode from a compute node to see available exports.</li><li>Monitor NFS performance with nfsstat -c (client) and nfsstat -s (server).</li><li>For better performance with large files, increase rsize and wsize (32768 or 65536).</li><li>If you see “Permission Denied” errors → check UID/GID consistency via LDAP and no_root_squash.</li><li>Stale mounts? Use sudo umount -f /mountpoint or restart autofs.</li><li>Always test with a normal user account, not just root.</li></ul><h3>What’s Next?</h3><p>In <strong>Part 5</strong>, we will install and configure LMOD, the environment modules system that allows users to easily load different versions of software (CUDA, Python, TensorFlow, etc.) without conflicts.</p><p>You now have centralized users (LDAP) and shared storage (NFS), now the cluster is starting to feel unified!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0cda80505b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 3…]]></title>
            <link>https://medium.com/@dsnikki07/building-a-high-performance-computing-hpc-cluster-from-scratch-a-step-by-step-guide-part-3-c3897ff08c1b?source=rss-0e0bab1bbe9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3897ff08c1b</guid>
            <category><![CDATA[ldap]]></category>
            <category><![CDATA[hpc]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[slurm]]></category>
            <category><![CDATA[munge]]></category>
            <dc:creator><![CDATA[Ganesh]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 02:10:52 GMT</pubDate>
            <atom:updated>2026-04-22T02:10:52.093Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 3: Authentication Setup with MUNGE and LDAP</strong></h3><p>In Part 2, we installed Red Hat Enterprise Linux 9 on all nodes and configured encrypted local scratch space using Stratis. Now we move to one of the most critical layers of any HPC cluster: <strong>secure and centralized authentication</strong>.</p><p>In this part, we will set up two essential systems:</p><ol><li><strong>MUNGE</strong> — Lightweight authentication service required by Slurm for secure communication between nodes.</li><li><strong>LDAP (389 Directory Server)</strong> + <strong>SSSD</strong> — Centralized user account management so users have consistent identities across the entire cluster.</li></ol><p>Why These Tools?</p><ul><li><strong>MUNGE</strong>: Provides fast, secure credential passing between nodes. Slurm uses it to verify that a job submission coming from the head node is legitimate.</li><li><strong>LDAP + SSSD</strong>: Allows us to manage users in one central place (on the head node) instead of creating local accounts on every single node. This ensures consistent UIDs/GIDs, home directories, and group memberships across all 12 nodes.</li></ul><p>We use <strong>389 Directory Server</strong> (open-source LDAP) as an interim solution that can later integrate with Red Hat IdM/FreeIPA or Active Directory.</p><h3>Part 3.1: MUNGE Setup</h3><h4>On the HeadNode (Management/Login Node)</h4><ol><li>Install MUNGE packages:</li></ol><pre>sudo dnf install munge munge-libs munge-devel -y</pre><p>2. Generate the MUNGE key:</p><pre>sudo /usr/sbin/create-munge-key -f</pre><p>3. Secure the key (very important):</p><pre>sudo chown munge:munge /etc/munge/munge.key <br>sudo chmod 400 /etc/munge/munge.key</pre><p>4. Enable and start the MUNGE service:</p><pre>sudo systemctl enable --now munge</pre><p>5. Verify it is working locally:</p><pre>munge -n | unmunge</pre><p>You should see STATUS: Success (0).</p><h4>On All Compute Nodes</h4><ol><li>Install MUNGE:</li></ol><pre>sudo dnf install munge munge-libs -y</pre><p>2. Copy the MUNGE key from the HeadNode to each compute node (replace headnode with your actual hostname or IP):</p><pre>sudo scp headnode:/etc/munge/munge.key /etc/munge/munge.key</pre><p>3. Set correct permissions on every compute node:</p><pre>sudo chown munge:munge /etc/munge/munge.key<br>sudo chmod 400 /etc/munge/munge.key</pre><p>4. Enable and start MUNGE:</p><pre>sudo systemctl enable --now munge</pre><p>5. Perform a cross-node test from the HeadNode (replace compute-01 with a real node name):</p><pre>munge -n | ssh compute-01 unmunge</pre><p>Success means the key is identical and network communication is allowed.</p><p><strong>Tip</strong>: Repeat the copy + permission steps on all 12 compute nodes. Automation with Ansible is recommended for larger clusters.</p><h3>Part 3.2: LDAP Setup (Centralized User Management)</h3><h4>On the HeadNode — Install and Configure 389 Directory Server</h4><ol><li>Install the LDAP server:</li></ol><pre>sudo dnf install -y 389-ds-base</pre><p>2. Create a configuration file (example: /root/ldap-config/headnode-ds.inf):</p><pre>sudo mkdir -p /root/ldap-config<br>sudo nano /root/ldap-config/headnode-ds.inf</pre><p>Paste the following content (modify full_machine_name and root_password):</p><pre>[general]<br>full_machine_name = headnode.yourdomain.local<br>start = True<br><br>[slapd]<br>db_lib = mdb<br>instance_name = ldap<br>ldapi = /run/slapd-ldap.socket<br>mdb_max_size = 20Gb<br>root_password = YourStrongDirectoryManagerPassword123!<br>self_sign_cert = True<br><br>[backend-userroot]<br>sample_entries = no<br>suffix = dc=cluster,dc=local<br>create_suffix_entry = True</pre><p>3. Create the instance:</p><pre>sudo dscreate from-file /root/ldap-config/headnode-ds.inf</pre><p>4. Check status:</p><pre>sudo systemctl status dirsrv@ldap</pre><h4>Define LDAP Structure</h4><p>Create a base structure file (/root/ldap-config/base.ldif):</p><pre>sudo nano /root/ldap-config/base.ldif</pre><pre>dn: ou=people,dc=cluster,dc=local<br>objectClass: organizationalUnit<br>ou: people<br><br>dn: ou=groups,dc=cluster,dc=local<br>objectClass: organizationalUnit<br>ou: groups<br><br>dn: ou=services,dc=cluster,dc=local<br>objectClass: organizationalUnit<br>ou: services</pre><p>Apply it:</p><pre>ldapadd -x -D &quot;cn=Directory Manager&quot; -W -f /root/ldap-config/base.ldif</pre><h4>Create SSSD Reader Service Account</h4><p>Create /root/ldap-config/sssd-reader.ldif:</p><pre>dn: cn=sssd-reader,ou=services,dc=cluster,dc=local<br>objectClass: top<br>objectClass: person<br>objectClass: organizationalPerson<br>objectClass: inetOrgPerson<br>cn: sssd-reader<br>sn: sssd-reader<br>userPassword: YourStrongServicePassword123!</pre><p>Add the account:</p><pre>ldapadd -x -D &quot;cn=Directory Manager&quot; -W -f /root/ldap-config/sssd-reader.ldif</pre><h3>Part 3.3: Configure SSSD on All Compute Nodes (and HeadNode)</h3><ol><li>Install required packages on every node:</li></ol><pre>sudo dnf install -y sssd sssd-ldap sssd-tools oddjob oddjob-mkhomedir authselect</pre><p>2. Create the SSSD configuration (/etc/sssd/sssd.conf):</p><pre>sudo nano /etc/sssd/sssd.conf</pre><pre>[sssd]<br>services = nss, pam<br>config_file_version = 2<br>domains = cluster.local<br><br>[domain/cluster.local]<br>id_provider = ldap<br>auth_provider = ldap<br>ldap_uri = ldap://headnode.yourdomain.local<br>ldap_search_base = dc=cluster,dc=local<br>ldap_default_bind_dn = cn=sssd-reader,ou=services,dc=cluster,dc=local<br>ldap_default_authtok_type = password<br>ldap_default_authtok = YourStrongServicePassword123!<br>cache_credentials = True<br>entry_cache_timeout = 600<br>entry_cache_negative_timeout = 60<br>ldap_user_object_class = posixAccount<br>ldap_group_object_class = posixGroup</pre><p>3. Secure the config file:</p><pre>sudo chown root:root /etc/sssd/sssd.conf<br>sudo chmod 600 /etc/sssd/sssd.conf</pre><p>4. Enable SSSD and automatic home directory creation:</p><pre>sudo authselect select sssd with-mkhomedir --force<br>sudo systemctl enable --now oddjobd sssd<br>sudo systemctl restart sssd</pre><p>Verify user lookup:</p><ol><li>Bash</li></ol><pre>getent passwd testuser     # Should return LDAP user if created<br>id username</pre><h3>Best Practices &amp; Important Notes</h3><ul><li>Always test with a dummy user before migrating real users.</li><li>Backup /etc/passwd, /etc/shadow, and /etc/group on the HeadNode before removing local accounts.</li><li>Use strong, unique passwords for Directory Manager and service accounts.</li><li>For production, consider enabling TLS encryption on LDAP (beyond self-signed certs).</li><li>Document all service passwords in a secure password manager.</li></ul><h3>What’s Next?</h3><p>In <strong>Part 4</strong>, we will configure <strong>NFS shared storage</strong> so that home directories, project spaces, and data are accessible from all nodes.</p><p>You now have secure node-to-node authentication (MUNGE) and centralized user management (LDAP + SSSD) — two foundational pillars of a professional HPC cluster.</p><p>If you encounter any issues (especially with LDAP binding or SSSD), share the exact error message in the comments and I’ll help troubleshoot.</p><p>Ready for Part 4? Let me know! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3897ff08c1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 2…]]></title>
            <link>https://medium.com/@dsnikki07/building-a-high-performance-computing-hpc-cluster-from-scratch-a-step-by-step-guide-part-2-233444882fb0?source=rss-0e0bab1bbe9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/233444882fb0</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[rhel]]></category>
            <category><![CDATA[hpc]]></category>
            <dc:creator><![CDATA[Ganesh]]></dc:creator>
            <pubDate>Fri, 17 Apr 2026 22:57:08 GMT</pubDate>
            <atom:updated>2026-04-17T22:58:57.027Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 2: Operating System Installation and Local Scratch Configuration</strong></p><p>In Part 1, we introduced the motivation for building an on-premises HPC cluster and outlined its architecture: 12 compute nodes divided into three tiers (High-Density GPU, Balanced GPU, and CPU-only), plus a central management/login node.</p><p>In this part, we begin the hands-on work: installing <strong>Red Hat Enterprise Linux 9</strong> on all nodes and configuring encrypted local scratch space using <strong>Stratis</strong>. This creates a stable, consistent foundation and provides each compute node with fast, private temporary storage for intermediate files, checkpoints, and large temporary datasets.</p><h3>Why RHEL 9 and Stratis?</h3><p><strong>Red Hat Enterprise Linux 9</strong> offers long-term support (up to 10 years with extended updates), excellent hardware compatibility, mature NVIDIA and Slurm ecosystems, and centralized patching via Red Hat Satellite. It is a safe, enterprise-grade choice for production HPC.</p><p><strong>Stratis</strong> is Red Hat’s modern storage management system. It provides thin provisioning, snapshots, encryption, and easy expansion — making it ideal for per-node scratch space that can grow as needed without complex manual LVM management. <br>Note: With Stratis, we have noticed it’s very slow with small file sizes. So choose the file system according to your use case.</p><h3>Prerequisites Before Starting</h3><ul><li>RHEL 9.4 (or latest 9.x) ISO downloaded from your organization’s mirror or Red Hat portal.</li><li>iDRAC (or equivalent remote management) credentials for every node.</li><li>Network details: static IP addresses, subnet, gateway, DNS, and hostname convention.</li><li>Activation key and organization ID for Red Hat Satellite registration.</li><li>Backup any existing data (though we assume nodes are freshly provisioned).</li><li>Strong passphrase for disk encryption (record it securely).</li></ul><h3>Step 1: Installing RHEL 9 via iDRAC Remote Console</h3><p>All nodes in this cluster support out-of-band management. We use the iDRAC virtual console for a clean, consistent installation.</p><ol><li>Log into the iDRAC web interface for the target node.</li><li>Launch the <strong>Virtual Console</strong>.</li><li>Mount the RHEL 9 ISO as virtual media:</li></ol><ul><li>Go to <strong>Virtual Media</strong> → <strong>Connect Virtual Media</strong> → Upload or select the RHEL ISO.</li></ul><p>4. Reboot the server and press <strong>F10</strong> (or the appropriate key shown on screen) during POST to enter the <strong>Lifecycle Controller</strong>.</p><p>5. In the Lifecycle Controller, select <strong>OS Deployment</strong> → Launch the RHEL installer.</p><p>6. In the graphical installer:</p><ul><li><strong>Language &amp; Keyboard</strong>: Choose your preference.</li><li><strong>Installation Destination</strong>: Select all drives you want to use. Enable <strong>Custom</strong> partitioning if needed, but for simplicity, use <strong>Automatic</strong> with encryption enabled.</li><li><strong>Disk Encryption</strong>: Enable full-disk encryption and set a strong passphrase. This protects data at rest.</li><li><strong>Network &amp; Hostname</strong>: Configure a static IP in your private cluster subnet, set the hostname (e.g., compute-01, gpu-high-01, cpu-01), and enable the interface.</li><li><strong>Software Selection</strong>: Choose <strong>Server with GUI</strong> (for initial setup) or <strong>Minimal Install</strong>. You can switch to minimal later if desired.</li><li><strong>User Creation</strong>: Create the admin user with sudo privileges and a strong password.</li><li><strong>Security Policy</strong>: Apply the default or a custom profile suitable for HPC.</li></ul><p>7. Click <strong>Begin Installation</strong>. The process typically takes 10–20 minutes.</p><p>8. After installation completes, remove the virtual media, reboot the node, and enter the encryption passphrase when prompted.</p><p><strong>Post-Installation First Boot Tasks</strong> (run as root or with sudo):</p><pre># Verify IP address and connectivity<br>hostname -I</pre><h3>Step 2: Registering with Red Hat Satellite</h3><p>Registering enables official repositories, security updates, and package management.</p><pre># Register the system (replace with your actual key and organization)<br>sudo rhc connect --activation-key YOUR_ACTIVATION_KEY_HERE \<br>                 --organization YOUR_ORGANIZATION_ID<br><br># Or using subscription-manager (alternative method)<br>sudo subscription-manager register --activationkey=YOUR_ACTIVATION_KEY \ <br>                                    --org=YOUR_ORGANIZATION_ID</pre><p>Verify registration:</p><pre>sudo subscription-manager status</pre><p>Update the system:</p><pre>sudo dnf update -y<br><br># Install kernel modules extra (important for NVIDIA and storage drivers)<br>sudo dnf install kernel-modules-extra-$(uname -r) -y</pre><h3>Step 3: Configuring Encrypted Local Scratch Space with Stratis</h3><p>Each compute node needs a fast, private scratch area (~8–10 TiB) for temporary files during jobs. We use Stratis for its modern features: encryption, thin provisioning, and simple management.</p><h4>Preparation (on every compute node)</h4><ol><li>Install Stratis packages:</li></ol><pre>sudo dnf install stratisd stratis-cli -y</pre><p>2. Enable and start the Stratis daemon:</p><pre>sudo systemctl enable --now stratisd</pre><p>3. Identify the drives for scratch space using lsblk:</p><pre>lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS</pre><ul><li>Look for the additional HDDs/SSDs (e.g., /dev/sdb and /dev/sdc). <strong>Be extremely careful</strong> — these commands will wipe data.</li></ul><p>4. Create a secure encryption keyfile (32 bytes is standard):</p><pre>sudo mkdir -p /root/keys <br>head -c 32 &lt;/dev/urandom &gt; /root/keys/scratch-keyfile <br>chmod 600 /root/keys/scratch-keyfile</pre><p>5. Load the key into the kernel keyring:</p><pre>stratis key set --keyfile-path /root/keys/scratch-keyfile scratch-key</pre><h4>Create the Stratis Pool and Filesystem</h4><pre># Create the encrypted pool using the identified drives<br># Replace /dev/sdX and /dev/sdY with your actual drive names<br>sudo stratis pool create --key-desc scratch-key scratch-pool /dev/sdX /dev/sdY</pre><pre># Disable overprovisioning (recommended when you have a single large filesystem)<br>sudo stratis pool overprovision scratch-pool no</pre><p>Create the filesystem (adjust size to your available capacity, e.g., 1.8 TiB):</p><pre>sudo stratis filesystem create --size 1.8TiB scratch-pool scratch-fs</pre><p>Mount it:</p><pre>sudo mkdir -p /scratch<br>sudo mount /dev/stratis/scratch-pool/scratch-fs /scratch</pre><h4>Make the Mount Persistent</h4><p>Edit /etc/fstab to ensure the scratch space mounts automatically after reboot:</p><pre>sudo nano /etc/fstab</pre><p>Add this line at the end:</p><pre>/dev/stratis/scratch-pool/scratch-fs   /scratch   xfs   defaults,x-systemd.requires=stratisd.service   0 0</pre><h4>Remount After Reboot (Test Procedure)</h4><pre># After reboot, unlock and mount<br>stratis key set --keyfile-path /root/keys/scratch-keyfile scratch-key<br>sudo stratis pool start --unlock-method keyring --name scratch-pool<br>sudo mount /dev/stratis/scratch-pool/scratch-fs /scratch<br><br># Verify<br>df -h | grep scratch<br>lsblk</pre><h3>Best Practices and Safety Notes</h3><ul><li><strong>Keyfile Security</strong>: Store a backup of /root/keys/scratch-keyfile in a secure, encrypted location (e.g., external drive or password manager). Losing this key means permanent data loss.</li><li><strong>Drive Identification</strong>: Always double-check lsblk before wiping drives. Consider labeling drives physically.</li><li><strong>Monitoring</strong>: Use stratis pool list, stratis filesystem list, and df -h regularly.</li><li><strong>Expansion</strong>: Stratis makes it easy to add more drives later with stratis pool add-data scratch-pool /dev/newdrive.</li><li><strong>Consistency</strong>: Apply the exact same Stratis configuration and key naming on all compute nodes for operational simplicity.</li></ul><h3>What’s Next?</h3><p>In <strong>Part 3</strong>, we will set up secure inter-node authentication with <strong>MUNGE</strong> and centralized user management using <strong>LDAP (389 Directory Server)</strong> + <strong>SSSD</strong>. This is critical before installing Slurm or NFS.</p><p>You now have a clean RHEL 9 installation with reliable local scratch on every compute node — a solid foundation for the rest of the cluster.</p><p>If you run into issues during installation or Stratis setup, feel free to describe the error in the comments.</p><p>See you in Part 3!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=233444882fb0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 1…]]></title>
            <link>https://medium.com/@dsnikki07/building-a-high-performance-computing-hpc-cluster-from-scratch-a-step-by-step-guide-part-1-2e76290552f6?source=rss-0e0bab1bbe9d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e76290552f6</guid>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[high-performace-computing]]></category>
            <category><![CDATA[rhel]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[hpc]]></category>
            <dc:creator><![CDATA[Ganesh]]></dc:creator>
            <pubDate>Sat, 14 Mar 2026 20:25:27 GMT</pubDate>
            <atom:updated>2026-03-14T20:25:27.735Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 1: Introduction, Motivation, and Cluster Architecture Overview</h3><p>High-Performance Computing (HPC) enables researchers, data scientists, engineers, and AI practitioners to solve problems that are simply impossible — or impractically slow — on ordinary workstations or even powerful personal servers. In late 2025 / early 2026, many organizations (universities, labs, small-to-medium research groups, startups, and specialized engineering teams) still choose to build <strong>on-premises HPC clusters</strong> rather than relying exclusively on public cloud offerings.</p><p>This multi-part series walks through the real-world design, installation, and configuration of a modest but production-ready GPU-accelerated HPC cluster built for a computational research environment. The system runs Red Hat Enterprise Linux 9, uses Slurm as the workload manager, NVIDIA professional GPUs for acceleration, NFS &amp; AutoFS(as an interim shared filesystem) combined with Stratis for local scratch, 389 Directory Server + SSSD for centralized identity, MUNGE for secure node-to-node authentication, and LMOD for environment modules.</p><h3>Why Build an On-Premises HPC Cluster in 2025–2026?</h3><p>Even with mature cloud HPC offerings (AWS ParallelCluster, Azure CycleCloud, Google Cloud HPC Toolkit, NVIDIA DGX Cloud, etc.), many teams still prefer on-premises infrastructure for one or more of the following reasons:</p><ul><li><strong>Predictable long-term cost for steady workloads:<br></strong> If your cluster runs at &gt;50–60% utilization year-round, the total cost of ownership (TCO) of owned hardware usually becomes lower than cloud after 2–4 years — especially once you factor in data egress fees, premium GPU instance pricing, and sustained high utilization.</li><li><strong>Data gravity and compliance / sovereignty requirements:<br></strong> Large datasets, protected health information (PHI under HIPAA), export-controlled data (ITAR/EAR), or GDPR-restricted workloads often cannot — or should not — leave the premises without complex (and expensive) arrangements.</li><li><strong>Full control over hardware stack and low-level tuning:<br></strong> You can choose exact CPU models, memory configuration, NVLink topology (if applicable), storage RAID levels, file system tuning, and driver versions without being limited to what cloud providers certify.</li><li><strong>Elimination of performance variability: </strong><br>No hypervisor overhead, no co-tenant interference, and deterministic behavior — critical for latency-sensitive MPI codes or tightly-coupled multi-GPU training.</li><li><strong>Hybrid-friendly stepping stone:<br></strong> A well-designed on-prem cluster can later burst to cloud during peak demand or for specific containerized / serverless workloads.</li></ul><p>Cloud HPC remains superior for bursty / exploratory work, rapid prototyping, or teams without CapEx budget or sysadmin capacity. Many production environments now use <strong>hybrid</strong> models — core steady-state work on-prem, burst / training runs in cloud.</p><h3>Cluster Design Goals</h3><p>This cluster was built with the following priorities:</p><ul><li>Rapidly enable GPU-accelerated workloads (deep learning, image analysis, scientific computing)</li><li>Provide substantial local scratch space per node for intermediate files and checkpoints</li><li>Deliver reliable shared storage for home directories, project data, and input/output</li><li>Centralize user identity and access control</li><li>Enforce fair resource allocation and GPU scheduling via Slurm</li><li>Keep initial deployment simple while designing for future growth (parallel file system, container support, advanced monitoring)</li></ul><h3>High-Level Cluster Architecture (12 Compute Nodes Total)</h3><p>All compute nodes are equipped with out-of-band remote management interfaces, enabling console access, power control, hardware monitoring, and firmware updates from a central location.</p><p>The cluster is logically divided into three tiers (which will map directly to Slurm partitions):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGmAmNnHX8rpimavN5T2BQ.png" /></figure><h3>Software Stack Decisions:</h3><ul><li><strong>Operating System</strong> → Red Hat Enterprise Linux 9 (long support lifecycle, mature NVIDIA &amp; Slurm ecosystem, satellite patching)</li><li><strong>Workload Manager</strong> → Slurm (dominant in research HPC, excellent GRES GPU scheduling)</li><li><strong>Environment Modules</strong> → LMOD (modern Lua-based, hierarchical, fast spider cache)</li><li><strong>Shared Storage </strong>→ Autofs &amp; NFSv4 (simple, reliable, quick to deploy)</li><li><strong>Local Scratch</strong> → Stratis + XFS (encryption, thin provisioning, snapshots, easy to expand)</li><li><strong>GPU Software</strong> → Official NVIDIA drivers + CUDA toolkit (via package manager)</li><li><strong>Authentication</strong> → MUNGE (required by Slurm) + LDAP/SSSD</li></ul><h3>Core Hardware Assumptions (Adapt to Your Budget)</h3><ul><li><strong>Head/login node</strong>: 2× Intel Xeon Gold / Platinum or AMD EPYC, 256–512 GB RAM, RAID-10 SSDs for OS + /home, large HDDs/SSDs for NFS shares</li><li><strong>Compute nodes</strong>: 2× Intel Xeon or AMD EPYC (56–64 cores each), 1–2 TB RAM, 1–4× NVIDIA L40 / A40 / RTX 6000 Ada / H100, NVMe SSDs for OS + Stratis scratch pool</li><li><strong>Networking</strong>: at minimum 10 GbE private VLAN; ideally 25/100 GbE or Mellanox/NVIDIA ConnectX InfiniBand for serious MPI / multi-GPU work</li><li><strong>Power &amp; cooling</strong>: appropriately rated PDU, UPS, good airflow — GPUs are power-hungry (~300–350 W each)</li></ul><h3>What’s Coming in the Series</h3><p><strong>Part 2</strong> — Operating system installation (RHEL 9 via remote console), initial hardening, hostname &amp; network configuration, and creation of encrypted local scratch space using Stratis on every compute node.</p><p>From Part 2 onward you will see complete, production-tested command sequences with detailed explanations of flags, safety considerations, and common pitfalls.</p><p>Feel free to ask questions or share your own tweaks in the comments!<br>Happy Clustering🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e76290552f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>