---
title: "Multi-Node LLM Training on Slurm: sbatch to Production"
thumbnail_title: "From sbatch to a production run"
description: "Learn how multi-node LLM training works on Slurm, from your first sbatch script to a production-ready run with shared storage, NCCL tests and checkpoints."
social_summary: "Getting a multi-node training job to start is the easy part. Here is what it takes for one to run for days without losing work: shared storage, a tested network, regular checkpoints and early warning on bad GPUs."
author: "hemant-singh-tanwar"
date: "October 26, 2026"
category: "AI & Machine Learning"
disclosure: "Disclosure: E2E Networks operates the TIR Training Cluster described in the closing section of this post."
faqAccordion: true
featured_image: "https://strapi-main-website.objectstore.e2enetworks.net/multi_node_llm_training_slurm_1f1ab39662.png"
---

Multi-node LLM training on Slurm means running one training job across several GPU servers at once. Slurm decides which servers to use and starts the work on all of them together. You describe the job in a short script, submit it with the `sbatch` command, and Slurm handles the rest.

Getting a job to start is the easy part. A run that can go on for days without losing work needs four more things: shared storage, a tested network between servers, regular checkpoints, and a way to catch failing hardware early. This post walks through each step in plain language, from your first script to a run you can trust.

## Why does LLM training need more than one GPU server?

Bigger LLMs do not fit on one server, and even smaller ones can take too long to train there.

Training needs far more memory than the model itself. Besides the model's weights, the GPUs must also hold gradients and optimizer state. With the usual mixed-precision setup and the Adam optimizer, that comes to about 16 bytes per parameter. That is before counting the working data for each batch.

So a 7-billion-parameter model needs roughly 112 GB just for this. A 70-billion-parameter model needs about 1.1 TB. A typical GPU server has 8 GPUs, and eight 80 GB GPUs give you 640 GB in total. The bigger model does not fit.

Splitting the work across servers solves both problems. Each server holds a share of the model or the data, and all the GPUs work in parallel. The catch is that they must stay in sync, swapping results after every training step. That is why the network between servers, and the software that coordinates them, matter so much.

## What is Slurm?

Slurm is free, open-source software that shares a group of computers among many users and jobs. It is widely used to schedule work on supercomputers and GPU clusters. By NVIDIA's own account, Slurm runs on more than half of the top 10 and top 100 systems in the TOP500 list of the world's fastest supercomputers.

Think of Slurm as the host at a busy restaurant. You tell the host how many seats you need and for how long. The host keeps a waiting list, seats your whole group together when enough tables are free, and clears the tables when you leave. Slurm does the same with GPUs. You ask for, say, 4 servers with 8 GPUs each for 24 hours, and Slurm starts your job only when all 32 GPUs are free at the same time.

That last part matters for training. A distributed job cannot start on half its GPUs and wait for the rest. Slurm starts every part together, or not at all.

Slurm is developed by SchedMD. NVIDIA acquired SchedMD in December 2025 and has said Slurm will stay open source and vendor-neutral.

### Slurm terms you will see

| Term | What it means |
|---|---|
| Node | One server in the cluster. In a training cluster, a worker node usually has 8 GPUs. |
| Job | A piece of work you ask Slurm to run, such as a training run. |
| Partition | A named queue that covers a set of nodes, with its own rules such as time limits. |
| `sbatch` | Submits a job script to the queue. You get your terminal back straight away. |
| `srun` | Starts a program on the nodes Slurm gave you. Inside a job script, it launches training on every node. |
| `squeue` | Shows the queue: which jobs are running and which are waiting. |
| `sinfo` | Shows the nodes and whether each is free, busy or out of service. |

## What does a Slurm training cluster look like?

A Slurm training cluster has five parts: a login node where you work, a controller that makes decisions, GPU worker nodes that do the training, a fast network between them, and storage that every node shares.

- **Login node.** You connect here over SSH to edit scripts and submit jobs. It is not meant for training. Heavy work here does not use the GPUs and can slow the node down for everyone.
- **Controller.** Runs the Slurm scheduler. It holds the queue, decides which job runs where, and keeps a history of past jobs.
- **Worker nodes.** The GPU servers that run your training. Slurm hands them out to jobs.
- **Network fabric.** A fast, low-delay network between workers, usually InfiniBand with RDMA. It lets GPUs on different servers swap data quickly, which multi-node training depends on.
- **Shared storage.** A file system mounted at the same path on the login node and every worker, so all nodes see the same code, data and checkpoints.

![Diagram of a Slurm training cluster. You connect to the login node over SSH to edit scripts and submit jobs, and sbatch sends the job to the controller, which holds the queue, picks the nodes and starts the job on every one of them. Four GPU worker nodes of 8 GPUs each are joined by an RDMA InfiniBand network on which the GPUs swap results after every step. Shared storage, holding code, data, container images, checkpoints and logs, is mounted at the same path on the login node and every worker.](https://strapi-main-website.objectstore.e2enetworks.net/slurm_training_cluster_diagram_bf0ef3896d.png)

*Slurm training cluster: login node, controller, four GPU workers and shared storage.*

The login node and every worker see the same shared storage, and the RDMA network links the workers to each other.

## From sbatch to first run: how a multi-node job works

A multi-node training job goes from a text file to 32 working GPUs in seven steps. The example below uses 4 nodes with 8 GPUs each.

1. **Connect to the login node.** You SSH in. This is where you prepare and submit everything.
2. **Put your files on shared storage.** Your code, dataset, container image and an empty checkpoint folder all go on the shared file system, so every node can reach them.
3. **Write a job script.** This is a short shell script. The `#SBATCH` lines at the top tell Slurm what you need: how many nodes, how many GPUs, how long, and where to write logs.
4. **Submit it with `sbatch`.** Slurm puts the job in the queue and gives you a job ID.
5. **Wait for resources.** The job stays pending until 4 full nodes are free at once. `squeue` shows why it is waiting, such as `Resources` (not enough free GPUs) or `Priority` (another job is ahead).
6. **Slurm starts the job on every node.** `srun` launches your container on all 4 nodes together. Inside each one, a launcher such as `torchrun` or DeepSpeed starts one training process per GPU, 32 in total.
7. **The GPUs train together.** After each step they share results over the network using NCCL, NVIDIA's library for GPU-to-GPU communication. Logs and checkpoints are written to shared storage as the run goes.

### What a job script looks like

Here is a simplified script for the example above. The table after it explains what each part does.

```bash
#!/bin/bash
#SBATCH --job-name=llm-pretrain
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=24:00:00
#SBATCH --output=/pfs/logs/%x-%j.out

export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
export MASTER_PORT=29500

# Keep NCCL on InfiniBand. Without these it can fall back to ordinary TCP.
export NCCL_IB_DISABLE=0
export NCCL_NET=IB
export NCCL_SOCKET_IFNAME=eth0
# export NCCL_IB_HCA==mlx5_0:1,mlx5_3:1   # pin the exact adapters on your nodes

srun --container-image=/pfs/images/pytorch.sqsh \
     --container-mounts=/pfs:/pfs,/dev/infiniband:/dev/infiniband \
     torchrun --nnodes=$SLURM_NNODES --nproc_per_node=8 \
              --rdzv_id=$SLURM_JOB_ID --rdzv_backend=c10d \
              --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
              /pfs/project/train.py --checkpoint-dir /pfs/checkpoints/run-01 --save-every 500
```

| Part of the script | What it does |
|---|---|
| `--nodes=4` | Asks for four GPU servers. |
| `--ntasks-per-node=1` | Starts one copy of the launcher on each server. The launcher then starts one process per GPU. |
| `--gpus-per-node=8` | Reserves all eight GPUs on each server. |
| `--exclusive` | Keeps other jobs off these servers. |
| `--mem=0` | Gives the job all the memory on each server. |
| `--time=24:00:00` | Stops the job after 24 hours. Always set a limit, or the job may inherit a partition's unlimited time. |
| `--output=...` | Writes the log to shared storage. `%x` becomes the job name and `%j` the job ID. |
| `MASTER_ADDR`, `MASTER_PORT` | Pick the first server as the meeting point where all 32 processes find each other at start-up. |
| `NCCL_IB_DISABLE`, `NCCL_NET`, `NCCL_SOCKET_IFNAME` | Keep GPU-to-GPU traffic on InfiniBand. Without them NCCL can fall back to TCP, which is far slower. |
| `srun --container-image` | Starts the same container on every server, so the software is identical everywhere. |
| `--container-mounts` | Makes shared storage and the InfiniBand devices visible inside the container. |
| `torchrun` | PyTorch's launcher. It starts 8 training processes per server and connects all 32. |
| `--save-every 500` | An option of the training script in this example: save a checkpoint every 500 steps. |

You submit it with one command, `sbatch train.sh`. Check on it with `squeue -u $USER` and follow the log with `tail -f`.

## What makes a run production-ready?

A production-ready run can survive a failed server, a missing file or a slow network without losing much progress. Six habits get you there.

### 1. Keep everything that matters on shared storage

Each node's local folders, such as `/tmp` and `/dev/shm`, are private to that node and get wiped on restart. Anything written inside a container is also thrown away when the job ends, unless that folder is mounted from shared storage. A run can look finished and still have saved nothing.

Put datasets, checkpoints, logs, container images and Python environments on the shared file system. Before a long run, write a test file from one node and read it from all the others.

### 2. Package your software in a container

Every node must run the same versions of Python, PyTorch, CUDA and NCCL. A container image makes sure of that. Import the image to shared storage once, then point every job at that file. This also stops dozens of nodes from downloading the same image at the same moment.

### 3. Test the network before the long run

When a multi-node job hangs, it is usually at the very start, while the GPUs are still trying to connect. Run a short NCCL all-reduce benchmark at your full node count before the real run.

Set `NCCL_DEBUG=INFO` and read the log. `NET/IB` means traffic is using InfiniBand. `NET/Socket` means NCCL fell back to ordinary TCP, which is far slower. The usual causes are InfiniBand devices not mounted into the container, wrong network adapter names in `NCCL_IB_HCA`, or a hard-coded start-up address. Our [Megatron-LM benchmark post](https://www.e2enetworks.com/blog/multi-node-megatron-lm-training-slurm-h200) shows what a correct NCCL log looks like on [H200 nodes](https://www.e2enetworks.com/gpus/nvidia-h200), including why `NCCL_IB_HCA` needs a leading equals sign.

### 4. Save checkpoints often, and make your script resume

A checkpoint is a saved snapshot of the model and optimizer that lets training continue from that point. A well-run cluster keeps going when one server fails, but the job on that server ends. Your progress is only as recent as your last checkpoint.

Save checkpoints to shared storage at a regular interval, for example every 500 steps, and keep the last few. Write your training script so it loads the latest checkpoint when it starts. Then a failed job only needs a restart.

### 5. Watch GPU health and take bad nodes out early

NVIDIA GPUs report hardware faults as XID errors. One faulty GPU can crash a job that spans 32 of them, sometimes hours in. When a node shows errors, drain it: Slurm lets current jobs finish and sends no new work there. Then reboot or repair it and return it to service.

You can automate part of this with a prolog, a script Slurm runs on each node before a job starts. If the script finds fewer GPUs than expected, it exits with an error and Slurm drains that node.

### 6. Monitor utilization and set limits

During training, GPU utilization should stay high and roughly even across GPUs. Near-zero utilization usually means the GPUs are waiting on data loading, or the code is running on the CPU. One GPU far below the rest can point to a stuck process.

Set alerts for low utilization and high temperature. Use partitions to keep short interactive jobs away from long training runs, and always set a `--time` limit.

## Pre-flight checklist for a long training run

Run through this list before you start a job that will take hours or days.

- Code, data, container image and checkpoint folder are on shared storage
- A test file written on one node can be read on every node
- A one-node test run shows the right number of GPUs and the right framework versions
- An NCCL all-reduce test at full scale shows `NET/IB`, not `NET/Socket`
- All nodes are healthy, with no XID errors and none drained
- The script saves checkpoints regularly and resumes from the latest one
- `--time` is set and logs go to shared storage
- Alerts are on for low GPU utilization and high temperature

## Running multi-node training on TIR Training Cluster

TIR Training Cluster, part of E2E Networks' [TIR AI platform](https://www.e2enetworks.com/tir), gives you a Slurm cluster with the pieces above already in place. It is a standard Slurm installation, so `sbatch`, `srun`, `squeue` and `sacct` work as described in this post. TIR installs and patches the scheduler for you. Under the hood, it runs on Kubernetes using SchedMD's Slinky project.

| What a production run needs | What TIR Training Cluster provides |
|---|---|
| Fast links between servers | GPU worker nodes connected by an RDMA fabric (InfiniBand) for NCCL traffic |
| Shared storage | [Parallel File System](https://www.e2enetworks.com/parallel-file-storage) and Shared File System, mounted at the same path on the login node and every worker |
| The same software on every node | Container jobs through Enroot and Pyxis: add `--container-image` to `srun` to run any Docker or OCI image, chosen per job |
| Catching bad hardware | A Nodes tab with per-GPU utilization, memory, temperature and power, XID error detection, and single-node reboot |
| Monitoring | A job queue, GPU and scheduler charts, and controller and worker logs in the browser, plus alerts |
| Automatic health checks | Prolog and epilog scripts for workers and the controller, set from the console |
| Sharing with a team | Up to 20 custom partitions, per-user SSH logins and private home directories |
| Growing the cluster | Add worker nodes to a running cluster without disturbing running jobs |
| Automation | A REST API for every console action |

The TIR documentation has a full walkthrough, including a 4-node PyTorch example, NCCL settings and a network test. Start with [Submit your first job](https://docs.e2enetworks.com/docs/tir/SlurmCluster/getting-started/first-job/), then move to [Multi-node container training](https://docs.e2enetworks.com/docs/tir/SlurmCluster/containers/multi-node-training/).

One thing to keep in mind: on TIR, a node failure does not take down the controller or the other nodes, but the job on the failed node still ends. Checkpointing stays your job, so the habits in this post apply here as on any Slurm cluster.

To get started, see [Model Training & Fine-tuning on TIR](https://www.e2enetworks.com/training-fine-tuning) or [talk to our team](https://www.e2enetworks.com/contact-sales).

## Frequently asked questions

### What is sbatch in Slurm?

`sbatch` submits a job script to Slurm's queue. The `#SBATCH` lines in the script say what the job needs, such as nodes, GPUs and a time limit. Slurm returns a job ID straight away and runs the job when resources are free, even if you log out.

### What is the difference between sbatch and srun?

`sbatch` puts a whole script in the queue to run later. `srun` starts a program on allocated nodes right away. Multi-node training uses both: `sbatch` submits the job, and an `srun` line inside the script launches training on every node at once.

### How many GPU nodes do I need to train an LLM?

It depends on the model's size and how fast you need results. As a rough guide, mixed-precision training with the Adam optimizer needs about 16 bytes of GPU memory per parameter, plus room for each batch. Add nodes until the model fits, then add more to finish sooner.

### Can I run Docker containers in a Slurm job?

Yes. Tools such as Enroot and Pyxis let Slurm run Docker or OCI images without root access. You add `--container-image` to your `srun` command, and the container starts on every allocated node with access to its GPUs.

### Why does my multi-node Slurm job hang at the start?

Most hangs happen while NCCL connects the GPUs. Run with `NCCL_DEBUG=INFO` to see where it stops. Common causes are InfiniBand devices not mounted in the container, wrong adapter names in `NCCL_IB_HCA`, or a hard-coded master address instead of one taken from the job's node list.

### Why is my Slurm job stuck in pending?

Slurm is waiting for resources that match your request. Run `squeue -u $USER` and read the `NODELIST(REASON)` column. `Resources` means not enough free GPUs, `Priority` means another job is ahead, and `PartitionTimeLimit` means your `--time` is longer than the partition allows.

### Does Slurm save my training progress if a node fails?

No. The cluster can keep running, but a job on a failed node ends. Your training script must save checkpoints to shared storage and resume from the latest one when you resubmit the job.

### Should I use Slurm or Kubernetes for LLM training?

Slurm is built for batch jobs that need many GPUs to start together, which suits large training runs. Kubernetes is stronger for long-running services such as model inference. Many teams train on Slurm and serve on Kubernetes, and some run Slurm on top of Kubernetes.