Multi-Node Megatron-LM Training on Slurm: 49% MFU on 16× H200

We ran a Nemotron-H-56B pretraining benchmark on an E2E Networks TIR Slurm cluster: 2 nodes, 16 NVIDIA H200 GPUs, 400G InfiniBand, and a 24-hour job with a checkpoint to the Parallel File System every hour. After tuning, the run held 487 to 489 TFLOP/s per GPU, 49% MFU in BF16, for 17 hours straight.

Everything here was run on a fresh TIR Slurm cluster. Below are the full sbatch script, the NCCL settings for TIR's H200 nodes, how we sized the checkpoint interval, the tuning that took MFU from 42% to 49%, charts from the first 18 hours of the 24-hour run, and the errors we hit on the way (with fixes). If you already run Megatron-LM elsewhere, Steps 1 and 3 cover the TIR-specific parts.

The run at a glance

Why run Megatron-LM on Slurm on TIR, and what you get

TIR's Slurm cluster gives you standard Slurm on E2E's H200 GPU nodes: you SSH into a login node, submit sbatch jobs, and Slurm places them on GPU workers with InfiniBand between them. The pieces you'll touch in this post, as described in the TIR Slurm cluster concepts docs:

Figure 1. Job layout on the TIR Slurm cluster: 2 nodes, 16 GPUs, TP=8 × DP=2.

The layout the script builds: one torchrun per worker with eight ranks, tensor parallelism across the eight GPUs of a node over NVLink, and the two nodes forming the data-parallel pair over InfiniBand.

On TIR, the container's root filesystem is read-only by default, so anything that writes to $HOME needs --container-writable. Node-local disk is wiped whenever a worker pod is recreated, so checkpoints, logs and the Megatron-LM source belong on the PFS.

Prerequisites

• A TIR Slurm cluster with at least 2 H200 workers (8 GPUs each) and a PFS volume attached. In this post the PFS is mounted at /pfs on the login node and both workers.
• SSH access to the login node.
• An NGC API key to pull nvcr.io/nvidia/pytorch:25.08-py3.
• Outbound access from the workers to github.com for the one-time Megatron-LM clone, or a pre-cloned copy on /pfs.

Step 1 ends with a quick check that both workers see their GPUs, the InfiniBand devices and the PFS from inside the container. It comes after the image import because pulling the image straight from nvcr.io on both workers needs more scratch space than a worker's default /tmp usually has.

Step 1: Set up shared storage and import the container image

Create the workspace on the PFS. Slurm opens the --output log file before the job script runs, so the log directory must exist before the first sbatch:

Next, turn the NVIDIA PyTorch image into a squashfs file on /pfs. Pyxis can pull straight from nvcr.io on every job, but with two nodes each pulling roughly 20 GB at job start, a local .sqsh is much faster and works without registry access at run time.

Add your NGC credentials once (the literal username is $oauthtoken, so keep the single quotes):

Then import on a worker, with Enroot's scratch space pointed at the PFS. Enroot reads the credentials file on the node where the import runs. Without Login User Management, the workers do not share the login node's home directory, so put the file where the worker can read it: for example, write it to /pfs/enroot-config/.credentials and add ENROOT_CONFIG_PATH=/pfs/enroot-config to the --export list below. Extracting and squashing the layers needs 2 to 3 times the final image size, more than a worker pod's default /tmp usually has:

Note the # between registry and image path. That is Enroot's URI syntax; a / there fails. Verify the image on both workers. You want two different hostnames, 8 GPUs on each, uverbs devices under /dev/infiniband inside the container, and the PFS mounted. Then clean up:

If /dev/infiniband is empty, stop here: NCCL will fall back to TCP sockets and the run will be much slower.

Step 2: The multi-node sbatch script for torchrun and Pyxis

The script prepares the Megatron-LM source on the PFS, sets the network and Transformer Engine environment, decides whether to resume, and launches pretrain_gpt.py on all 16 GPUs. Every tunable is an environment variable with a default, so each tuning run in the Results section is a single sbatch --export command.

With PP=1, TP=8 is required. At TP=4 the 56B model's bf16 weights, fp32 gradients and the sharded optimizer state come to roughly 126 GB per GPU before activations, which does not fit on a 141 GB H200 with 8192-token sequences.

The complete script, saved as /pfs/nemotron-h/nemotron_h_56b_2x8h200.sbatch:

A few lines in the script need explaining:

• --container-writable. TIR's Pyxis configuration mounts the container root filesystem read-only. Triton and Transformer Engine compile kernels into ~/.triton at import time, so without this flag every rank dies before training starts.
• make instead of pip install -e .. The Megatron helper .so is the only thing that needs building. make -C megatron/core/datasets writes it into the source tree on /pfs; PYTHONPATH handles imports. This also avoids pip trying to write to /root/.local.
• The conditional --load. Megatron core_v0.15.2 raises is not a MLM or NeMo sharded checkpoint if --load points at an empty directory. Passing it only when latest_checkpointed_iteration.txt exists makes the same script work for both the first run and every resume.

Step 3: Pin NCCL to the 400G InfiniBand rails

Each TIR H200 node exposes 12 Mellanox devices, and only 8 of them are the InfiniBand compute rails. ibv_devinfo inside the container shows the split:

Left to its own device selection, NCCL may pick up the Ethernet ports as RoCE and mix them into the rings. Pin it to the eight IB rails with an exact-match list:

The leading = after NCCL_IB_HCA= matters. Without it, NCCL treats each entry as a prefix: an entry of mlx5_1 would also select mlx5_10 and mlx5_11. With =, every name is an exact match, so the list selects exactly these eight ports, and :1 pins port 1 on each card.

Run the first job with NCCL_DEBUG=INFO and look for one line per rank. This is what a correct setup prints on TIR:

Eight entries, all /IB, none /RoCE. Cross-node traffic goes over GDRDMA (GPUDirect RDMA, each GPU on its own rail), intra-node traffic over NVLink P2P/CUMEM, and the HPC-X plugin reports SHARP support. If you see NET/Socket instead, the InfiniBand devices are not visible inside the container. Device numbering can differ between nodes, so check ibv_devinfo -l on every node before trusting the list.

Step 4: Checkpoint Megatron-LM every hour inside a 24-hour Slurm job

Megatron-LM has no wall-clock checkpoint option; --save-interval counts iterations. Getting an hourly cadence inside a 24-hour Slurm job means setting the save interval, the retain interval and the exit time together.

Calibrate the interval from the measured step time. After 30 to 50 iterations, read elapsed time per iteration (ms) from the log and set SAVE_INTERVAL = floor(3600 / seconds_per_iteration). We calibrated on an early 25.5 s per iteration (3600 / 25.5 ≈ 141) and used 140; the run then settled at 24.5 s, so saves landed every 57 to 58 minutes rather than exactly 60. That is close enough for an hourly policy. Recalibrate whenever you change batch size, micro batch or precision: at the untuned 7.1 s per iteration and batch 16, the same hour was about 500 iterations.

Keep the retain interval a multiple of the save interval. --save-retain-interval deletes every checkpoint except the latest and those at multiples of its value. Megatron requires the retain interval to be a multiple of the save interval and stops at startup otherwise. The script derives it as SAVE_INTERVAL * 6 = 840, so one checkpoint every 6 hours survives the run: at about 3,300 iterations a day, that is iterations 840, 1,680 and 2,520, plus the latest.

Exit cleanly before Slurm's time limit. --exit-duration-in-mins 1410 makes Megatron write a final checkpoint and exit 23 h 30 m after training starts, about half an hour before the 24-hour --time. Slurm's default KillWait is 30 seconds, far too short for a checkpoint of this size, so relying on SIGTERM at the limit would lose up to an hour of work. About 3,300 iterations fit in the window at 25.5 s each.

Resuming is then a resubmit of the same script. It finds latest_checkpointed_iteration.txt, adds --load, and continues from that iteration.

Budget the storage before the first run. A 56B model with the distributed optimizer's fp32 state writes on the order of 750 to 800 GB per checkpoint. --async-save hides most of the pause, but each write still has to finish within the hour, which needs about 225 MB/s of sustained write bandwidth. Three retained checkpoints, the latest, and briefly the one being replaced come to about 3 to 4 TB over a 24-hour job. TIR's Parallel File System handles this comfortably; a single NFS server generally does not.

Step 5: Submit the job, monitor GPU utilisation and verify the first checkpoint

Submit from the login node. For the first run, keep NCCL_DEBUG=INFO on so the transport is in the log; drop it afterwards, it adds thousands of lines.

Each phase of the log has a line that tells you it worked:

The TIR dashboard's Jobs tab mirrors squeue. Once training starts, the Monitoring tab should show average GPU utilisation near 100% on both nodes; the Nodes tab has the per-GPU view. If both nodes stay at 0% several minutes after the job starts, the rendezvous has probably not completed; see the last row of the troubleshooting table.

After about an hour, confirm the first checkpoint landed:

With --async-save, the iter_* directory appears first and fills in over the next few minutes; the successfully saved checkpoint line marks the write finishing. To test resume, scancel the job after that line and resubmit: the log should show /pfs/nemotron-h/checkpoints at iteration 140. When the 24-hour job ends at 23 h 30 m, the same resubmit continues the run from its final checkpoint; uncomment --dependency=singleton in the header to queue the next day's job in advance.

Results: 489 TFLOP/s per GPU, 49% MFU

The tuned configuration settles at 24.5 s per iteration and 487 to 489 TFLOP/s per GPU on 16 H200s, which is 49.2 to 49.4% MFU against the H200's 989 TFLOP/s dense BF16 peak. The first working run, before tuning, did 420 TFLOP/s (42.4%). The tuned run differs from it in global batch size, TP communication overlap and per-step diagnostics. The tuned column below is the steady state of the 24-hour run (iterations 400 to 2100); the first-run column is the average of iterations 30 to 60 of that job:

Common to both: Nemotron-H-56B, 56.5B parameters (79 layers, hidden 8192, FFN 32768, GQA 64/8), sequence 8192, BF16, TP=8, PP=1, DP=2, cross-node traffic over 8× 400G IB with GPUDirect RDMA.

Megatron's throughput per GPU is model FLOPs only, with no activation recompute in this config, so it is directly comparable to peak. A first-principles check agrees: about 364 GFLOP per token (6 × 55.4B for the non-embedding and output layers, plus about 32 GFLOP of causal attention at sequence 8192) × 524,288 tokens per step ÷ 24.5 s ÷ 16 GPUs ≈ 487 TFLOP/s.

We changed all three settings together, so what follows is our reading of where the 7 points came from, not a measured breakdown. --tp-comm-overlap hides the per-layer all-gather and reduce-scatter across the 8 TP ranks behind the GEMMs. A global batch of 64 gives each DP rank 32 microbatches per step instead of 8, so the optimizer step and gradient reduction are amortised 4×. Dropping --log-params-norm, --log-num-zeros-in-grad and --log-straggler removes an extra pass over the parameters and gradients every step. Step time grew 3.5× for 4× the tokens, which is where the fixed per-step cost shows up. The comparison also overstates the tuning gain somewhat: the first-run figure comes from its first few minutes, while the tuned figure is from after the two-hour ramp, and the tuned run's own early iterations were around 470 TFLOP/s.

Micro batch 2 is the first thing to try next. It doubles the GEMM M dimension and, from the measured 100 GB peak at micro batch 1, should land near 120 GB per GPU (add --use-precision-aware-optimizer if it does not fit). FlashAttention-3, which the container does not ship, is usually faster than the FlashAttention-2 kernels on Hopper, and cuDNN fused attention (ATTN_BACKEND=fused) is worth a comparison too. TP=4 with PP=2 halves the participants in every TP collective at the cost of a small pipeline bubble. FP8 (FP8=1) should raise throughput further, but we have not measured it on this setup, and its MFU is quoted against the FP8 peak, so it is not comparable to these BF16 numbers. Most of these are a single sbatch --export setting for the script above; test them one at a time over 80 iterations and compare the iteration 20 to 30 throughput.

The 24-hour run

With the tuned defaults locked in, we submitted the 24-hour job at 10:35 on 29 September. The two charts below cover the first 18 hours of its log: throughput per GPU, and the checkpoint cadence.

Figure 2. Megatron-reported throughput per GPU. Source: Slurm job log, iterations 10–2,700, the first 18 hours of the 24-hour run, 29–30 Sep 2026.

After a two-hour ramp from 469 to the high 480s, throughput sat between 487 and 489 TFLOP/s per GPU for 17 hours. Some of the hourly checkpoints show up as a one-sample dip of up to about 8 TFLOP/s. From iteration 2160 the run stepped down to about 466 TFLOP/s and stayed there. Nothing in the job configuration changed at that point, so the likely causes are outside the training loop: GPU clock or power throttling on one node, or a degraded InfiniBand link. The filesystem is an unlikely cause: the run reads synthetic data, and each checkpoint pauses training for only about 2.6 s. The TIR Monitoring tab records per-node GPU power and temperature, and the Nodes tab shows DCGM metrics and XID errors, so those are the first places to check, along with nvidia-smi -q -d PERFORMANCE on both nodes.

Figure 3. Training pause at each checkpoint save, and which saves were retained. Source: save-checkpoint timer and retention messages in the Slurm job log, 19 saves from 11:35 on 29 Sep to 05:05 on 30 Sep.

Saves landed every 57 to 60 minutes (140 iterations at 24.4 to 25.6 s each). With --async-save, the training loop pauses only long enough to hand the sharded state to the background writer: about 2.6 s per save, roughly 0.1% of the hour. The exception was the first save, which took 46.6 s because PyTorch compiled the checkpoint extension on the fly; setting TORCH_CUDA_ARCH_LIST=9.0 in the script should shorten that one-time compile. Retention matched the arithmetic: after each save, the previous one was deleted unless its iteration was a multiple of 840, so iterations 840, 1680 and 2520 survived while the rest were removed within the hour.

At TP=8, the same arithmetic as the TP=4 estimate gives roughly 80 to 85 GB per GPU for weights, gradients and optimizer state. Megatron reported 81.6 GB allocated and a 100.4 GB peak per GPU (102.3 GB reserved) out of 141 GB, with activations accounting for about 19 GB of the peak.

Troubleshooting: the Megatron-LM and Pyxis errors we hit

Our first jobs failed within two minutes of starting, mostly because of how Pyxis sets up the container. All the fixes are already in the script; the table shows what each failure looks like in the log and how to fix it.

You can ignore Failed to create unified memory mempool and TORCH_NCCL_AVOID_RECORD_STREAMS is the default now; both appear on every rank and do not affect the run. A grad norm spike in the first hundred iterations (we saw 91 at iteration 60 after values around 1) is expected with only a 5-iteration warm-up and is clipped by --clip-grad 1.0; raise --lr-warmup-iters to a few hundred for a production run.

From benchmark to real training

This job trains Nemotron-H-56B from scratch on Megatron-LM's built-in synthetic token stream, so the numbers above measure the cluster, not a data pipeline. To run it on your own data, replace --mock-data and --tokenizer-type NullTokenizer with a --data-path blend of tokenized Megatron .bin/.idx files and the matching tokenizer. Raise --lr-warmup-iters from 5 to a few hundred and pick a --global-batch-size that suits your token budget. Set --train-iters and --lr-decay-iters for the run you actually intend, then recalibrate SAVE_INTERVAL from the new step time. Throughput will be within a few percent of what you see here, since the data pipeline is not the bottleneck at this scale.

To scale out, set --nodes=4 on a 4-node TIR cluster. That gives DP=4 with the same TP=8, and the c10d rendezvous and the IB pinning carry over unchanged. Recalibrate SAVE_INTERVAL, because the step time roughly halves and 140 iterations would then take about 30 minutes. With the global batch still at 64, each DP rank gets 16 microbatches instead of 32, so consider raising --global-batch-size. Test a resume from a 2-node checkpoint before relying on it.

Run your own training job on TIR

The cluster in this post is available to rent by the hour, with the same H200 nodes, 400G InfiniBand fabric and Parallel File System. Explore training and fine-tuning on TIR to size a cluster for your own run, or talk to our team about multi-node capacity.

Sources

• TIR Slurm cluster concepts, E2E Networks docs
• TIR Training Cluster: running training jobs, E2E Networks docs
• Megatron-LM core_v0.15.2, NVIDIA
• NVIDIA PyTorch container 25.08, NGC
• NCCL environment variables, NVIDIA docs
• Pyxis usage, NVIDIA on GitHub