---
title: "Multi-Node Megatron-LM Training on Slurm: 49% MFU on 16× H200"
description: "A complete multi-node Megatron-LM setup on a Slurm GPU cluster: the full sbatch script, NCCL pinned to 400G InfiniBand, hourly checkpointing, and 487–489 TFLOP/s per GPU (49% MFU) on 16× NVIDIA H200."
social_summary: "We pretrained a 56B-parameter model on 16× H200 for 24 hours and held 487–489 TFLOP/s per GPU. Here is the sbatch script, the NCCL settings, the checkpoint maths and every error we hit."
author: "jagmeet-singh"
date: "October 1, 2026"
category: "AI & Machine Learning"
disclosure: "Disclosure: E2E Networks operates the TIR Slurm cluster and the H200 GPU capacity this benchmark was run on."
faqAccordion: true
featured_image: "https://strapi-main-website.objectstore.e2enetworks.net/multi_node_megatron_lm_training_slurm_h200_573ed1e3dd.png"
---

We ran a Nemotron-H-56B pretraining benchmark on an E2E Networks TIR Slurm cluster: 2 nodes, 16 [NVIDIA H200 GPUs](https://www.e2enetworks.com/gpus/nvidia-h200), 400G InfiniBand, and a 24-hour job with a checkpoint to the [Parallel File System](https://www.e2enetworks.com/parallel-file-storage) every hour. After tuning, the run held 487 to 489 TFLOP/s per GPU, 49% MFU in BF16, for 17 hours straight.

Everything here was run on a fresh TIR Slurm cluster. Below are the full sbatch script, the NCCL settings for TIR's H200 nodes, how we sized the checkpoint interval, the tuning that took MFU from 42% to 49%, charts from the first 18 hours of the 24-hour run, and the errors we hit on the way (with fixes). If you already run Megatron-LM elsewhere, Steps 1 and 3 cover the TIR-specific parts.

## The run at a glance

| Parameter | Value |
|---|---|
| **Model** | Nemotron-H-56B configuration, 56.5B parameters — 79 layers, hidden 8192, FFN 32768, GQA 64/8, sequence 8192 |
| **Framework** | Megatron-LM `core_v0.15.2`, NVIDIA PyTorch container `25.08-py3` |
| **Hardware** | 2 nodes × 8 NVIDIA H200 (141 GB) = 16 GPUs |
| **Interconnect** | NVLink within a node; 8× 400G ConnectX-7 InfiniBand rails per node with GPUDirect RDMA between nodes |
| **Parallelism** | TP=8, PP=1, DP=2; micro batch 1, global batch 64 |
| **Precision** | BF16 |
| **Storage** | Parallel File System at `/pfs` — container image, source, checkpoints and logs |
| **Throughput** | 487–489 TFLOP/s per GPU, 21.4k tokens/s cluster-wide |
| **MFU** | 49.2–49.4% against the H200's 989 TFLOP/s dense BF16 peak |
| **Checkpoints** | ~800 GB each, written hourly, ~2.6 s training stall per save |
| **On-demand cost** | ₹8,040.80 per hour for all 16 GPUs (₹502.55/GPU-hour) |

## Why run Megatron-LM on Slurm on TIR, and what you get

TIR's Slurm cluster gives you standard Slurm on E2E's H200 GPU nodes: you SSH into a login node, submit sbatch jobs, and Slurm places them on GPU workers with InfiniBand between them. The pieces you'll touch in this post, as described in the [TIR Slurm cluster concepts docs](https://docs.e2enetworks.com/docs/tir/SlurmCluster/getting-started/slurm-cluster-concepts/):

| Piece | What it is on TIR | How this post uses it |
|---|---|---|
| Login node | The host you SSH into; runs `sbatch`, `squeue`, `scontrol` | Submitting and monitoring the job |
| Workers | GPU nodes, 8× H200 each, in the `all` partition | `--nodes=2 --gres=gpu:8` |
| Partition | Named group of workers (`all` by default) | Default partition, nothing to set |
| Parallel File System (PFS) | High-throughput shared storage, mounted at the same path on every node | `/pfs` for image, code, checkpoints, logs |
| Pyxis + Enroot | Slurm plugin that runs each job step inside a container image | `srun --container-image=...` |
| InfiniBand | 8× 400G ConnectX-7 rails per node, plus separate 200G Ethernet ports | NCCL pinned to the 8 IB rails |
| Jobs / Monitoring / Nodes tabs | Cluster dashboard views of `squeue`, GPU utilisation, node health | Watching the run without SSH |

![Job layout on the TIR Slurm cluster. A login node running sbatch, squeue and tail submits to the Slurm controller, which uses partition all with Pyxis and Enroot. Two worker nodes, slinky-0 and slinky-1, each run one torchrun with 8 ranks in a Pyxis container loaded from /pfs, each holding 8 H200 GPUs of 141 GB on NVLink at TP=8 and 8 ConnectX-7 400G InfiniBand rails, one per GPU. The two nodes exchange DP=2 gradient all-reduce traffic over InfiniBand with GPUDirect RDMA on all 8 rails. Both nodes read and write the same Parallel File System at /pfs, holding the pytorch-25.08 squashfs image, the megatron-lm source, hourly checkpoints of about 800 GB each, and logs.](https://strapi-main-website.objectstore.e2enetworks.net/training1_b843e69618.png)

*Figure 1. Job layout on the TIR Slurm cluster: 2 nodes, 16 GPUs, TP=8 × DP=2.*

The layout the script builds: one torchrun per worker with eight ranks, tensor parallelism across the eight GPUs of a node over NVLink, and the two nodes forming the data-parallel pair over InfiniBand.

On TIR, the container's root filesystem is read-only by default, so anything that writes to `$HOME` needs `--container-writable`. Node-local disk is wiped whenever a worker pod is recreated, so checkpoints, logs and the Megatron-LM source belong on the PFS.

## Prerequisites

- A TIR Slurm cluster with at least 2 H200 workers (8 GPUs each) and a PFS volume attached. In this post the PFS is mounted at `/pfs` on the login node and both workers.
- SSH access to the login node.
- An [NGC API key](https://org.ngc.nvidia.com/setup/api-key) to pull `nvcr.io/nvidia/pytorch:25.08-py3`.
- Outbound access from the workers to github.com for the one-time Megatron-LM clone, or a pre-cloned copy on `/pfs`.

Step 1 ends with a quick check that both workers see their GPUs, the InfiniBand devices and the PFS from inside the container. It comes after the image import because pulling the image straight from nvcr.io on both workers needs more scratch space than a worker's default `/tmp` usually has.

## Step 1: Set up shared storage and import the container image

Create the workspace on the PFS. Slurm opens the `--output` log file before the job script runs, so the log directory must exist before the first `sbatch`:

```bash
mkdir -p /pfs/nemotron-h/logs /pfs/images /pfs/enroot-tmp
```

Next, turn the NVIDIA PyTorch image into a squashfs file on `/pfs`. Pyxis can pull straight from nvcr.io on every job, but with two nodes each pulling roughly 20 GB at job start, a local `.sqsh` is much faster and works without registry access at run time.

Add your NGC credentials once (the literal username is `$oauthtoken`, so keep the single quotes):

```bash
mkdir -p ~/.config/enroot
echo 'machine nvcr.io login $oauthtoken password <YOUR_NGC_API_KEY>' >> ~/.config/enroot/.credentials
chmod 600 ~/.config/enroot/.credentials
```

Then import on a worker, with Enroot's scratch space pointed at the PFS. Enroot reads the credentials file on the node where the import runs. Without Login User Management, the workers do not share the login node's home directory, so put the file where the worker can read it: for example, write it to `/pfs/enroot-config/.credentials` and add `ENROOT_CONFIG_PATH=/pfs/enroot-config` to the `--export` list below. Extracting and squashing the layers needs 2 to 3 times the final image size, more than a worker pod's default `/tmp` usually has:

```bash
srun -N1 --export=ALL,ENROOT_TEMP_PATH=/pfs/enroot-tmp,ENROOT_CACHE_PATH=/pfs/enroot-tmp \
  enroot import -o /pfs/images/pytorch-25.08-py3.sqsh \
  docker://nvcr.io#nvidia/pytorch:25.08-py3
```

Note the `#` between registry and image path. That is Enroot's URI syntax; a `/` there fails. Verify the image on both workers. You want two different hostnames, 8 GPUs on each, uverbs devices under `/dev/infiniband` inside the container, and the PFS mounted. Then clean up:

```bash
srun -N2 --ntasks-per-node=1 --gres=gpu:8 \
  --container-image=/pfs/images/pytorch-25.08-py3.sqsh \
  --container-mounts=/pfs:/pfs \
  bash -c 'hostname; nvidia-smi -L | wc -l; ls /dev/infiniband; ls /pfs'

rm -rf /pfs/enroot-tmp
```

If `/dev/infiniband` is empty, stop here: NCCL will fall back to TCP sockets and the run will be much slower.

## Step 2: The multi-node sbatch script for torchrun and Pyxis

The script prepares the Megatron-LM source on the PFS, sets the network and Transformer Engine environment, decides whether to resume, and launches `pretrain_gpt.py` on all 16 GPUs. Every tunable is an environment variable with a default, so each tuning run in the Results section is a single `sbatch --export` command.

| Part of the script | What it does |
|---|---|
| `#SBATCH` header | 2 nodes, 1 task per node, 8 GPUs per node, exclusive, 24 h, log on `/pfs` |
| Paths + `CONTAINER_ARGS` | Everything under `/pfs/nemotron-h`; Pyxis runs the local `.sqsh` with `/pfs` mounted and a writable rootfs |
| One-time prep `srun` | Clones Megatron-LM `core_v0.15.2` and builds the dataset helper `.so`, on one node only |
| Rendezvous + environment | `MASTER_ADDR` from the node list, `PYTHONPATH`, node-local JIT caches, TE/UCX settings |
| Networking | `NCCL_IB_HCA` pinned to the 8 InfiniBand rails (Step 3) |
| Parallelism / batch | TP=8, PP=1 (DP=2), micro batch 1, global batch 64 |
| Duration / checkpointing | 23.5 h exit, save every 140 iterations (hourly), retain one every 6 h (Step 4) |
| Performance knobs | TP communication overlap on, FP8 off, per-step diagnostics off |
| `MEGATRON_ARGS` | The Nemotron-H-56B architecture and optimizer settings |
| Resume check | Adds `--load` only when a checkpoint tracker file exists |
| Launch | One torchrun per node with c10d rendezvous, 8 ranks each |

With PP=1, TP=8 is required. At TP=4 the 56B model's bf16 weights, fp32 gradients and the sharded optimizer state come to roughly 126 GB per GPU before activations, which does not fit on a 141 GB H200 with 8192-token sequences.

The complete script, saved as `/pfs/nemotron-h/nemotron_h_56b_2x8h200.sbatch`:

```bash
#!/bin/bash
#SBATCH --job-name=nemotron-h-56b
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1            # one torchrun per node, torchrun spawns 8 ranks
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=24:00:00
#SBATCH --output=/pfs/nemotron-h/logs/%x-%j.out
##SBATCH --dependency=singleton        # uncomment to chain resubmits of the same job name

set -euo pipefail

# ---------------- Paths (PFS, mounted at the same path on both nodes) ----------------
WORKSPACE_DIR=${WORKSPACE_DIR:-/pfs/nemotron-h}
MEGATRON_PATH=${MEGATRON_PATH:-${WORKSPACE_DIR}/megatron-lm}
CHECKPOINT_DIR=${CHECKPOINT_DIR:-${WORKSPACE_DIR}/checkpoints}
TENSORBOARD_DIR=${TENSORBOARD_DIR:-${WORKSPACE_DIR}/tensorboard}
DATA_CACHE_DIR=${WORKSPACE_DIR}/data_cache
IMAGE=${IMAGE:-/pfs/images/pytorch-25.08-py3.sqsh}   # enroot import of nvcr.io/nvidia/pytorch:25.08-py3
mkdir -p "$CHECKPOINT_DIR" "$TENSORBOARD_DIR" "$DATA_CACHE_DIR" "${WORKSPACE_DIR}/logs"

CONTAINER_ARGS=(
  --container-image="$IMAGE"
  --container-mounts="${WORKSPACE_DIR}:${WORKSPACE_DIR}"
  --container-workdir="$WORKSPACE_DIR"
  --no-container-mount-home
  --container-writable                 # rootfs is read-only by default on TIR; Triton/TE write to ~/.triton
)

# ---------------- One-time prep: Megatron-LM source on /pfs ----------------
# Runs on one node only so both nodes don't race building into the same shared dir.
srun --nodes=1 --ntasks=1 "${CONTAINER_ARGS[@]}" bash -c "
  set -ex
  if [ ! -d ${MEGATRON_PATH}/.git ]; then
    git clone --depth 1 -b core_v0.15.2 https://github.com/NVIDIA/Megatron-LM.git ${MEGATRON_PATH}
  fi
  if ! ls ${MEGATRON_PATH}/megatron/core/datasets/helpers_cpp*.so >/dev/null 2>&1; then
    make -C ${MEGATRON_PATH}/megatron/core/datasets   # helper .so only; PYTHONPATH handles imports
  fi
"

# ---------------- Rendezvous ----------------
export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)
export MASTER_PORT=29500
export PYTHONPATH=${MEGATRON_PATH}
export TRITON_CACHE_DIR=/tmp/triton-cache          # node-local JIT caches, not on /pfs
export TORCHINDUCTOR_CACHE_DIR=/tmp/inductor-cache

# ---------------- Transformer Engine / PyTorch / UCX environment ----------------
export NVTE_FWD_LAYERNORM_SM_MARGIN=16
export NVTE_BWD_LAYERNORM_SM_MARGIN=16
ATTN_BACKEND=${ATTN_BACKEND:-flash}                 # flash (FlashAttention-2) or fused (cuDNN)
if [ "$ATTN_BACKEND" = "fused" ]; then export NVTE_FUSED_ATTN=1; else export NVTE_FUSED_ATTN=0; fi
export TORCHINDUCTOR_WORKER_START=fork
export CUDA_DEVICE_MAX_CONNECTIONS=1
export TORCH_NCCL_AVOID_RECORD_STREAMS=1
export TORCH_NCCL_HIGH_PRIORITY=1
export UCX_MEM_MMAP_HOOK_MODE=none
export UCX_MEM_CUDA_HOOK_MODE=none
export UCX_MEM_MALLOC_HOOKS=n
export UCX_ERROR_SIGNALS=""

# ---------------- Networking: 8x 400G InfiniBand rails only (see Step 3) ----------------
export NCCL_IB_HCA=${NCCL_IB_HCA:-=mlx5_0:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_9:1,mlx5_10:1,mlx5_11:1}
export NCCL_SOCKET_IFNAME=${NCCL_SOCKET_IFNAME:-eth0}
export GLOO_SOCKET_IFNAME=${NCCL_SOCKET_IFNAME}

# ---------------- Parallelism / batch ----------------
GPUS_PER_NODE=8
TOTAL_GPUS=$((SLURM_NNODES * GPUS_PER_NODE))           # 16
TP=${TENSOR_PARALLELISM:-8}                             # TP stays on NVLink; TP=4 with PP=1 does not fit 141 GB
PP=${PIPELINE_PARALLELISM:-1}                           # DP = 16 / (8*1) = 2
MBS=${MICRO_BATCH_SIZE:-1}
GBS=${GLOBAL_BATCH_SIZE:-64}                            # 32 microbatches per DP rank

# ---------------- Duration / checkpointing (24 h run, hourly checkpoints; see Step 4) ----------------
TRAIN_ITERS=${TRAIN_ITERS:-100000}
EXIT_DURATION_MINS=${EXIT_DURATION_MINS:-1410}           # stop at 23 h 30 m with a final save
SAVE_INTERVAL=${SAVE_INTERVAL:-140}                      # 3600 / 25.5 s per iter = 141, rounded down to 140 -> 59.5 min
SAVE_RETAIN_INTERVAL=${SAVE_RETAIN_INTERVAL:-$((SAVE_INTERVAL * 6))}   # keep one every ~6 h

# ---------------- Performance knobs ----------------
TP_COMM_OVERLAP=${TP_COMM_OVERLAP:-1}   # hide TP all-gather/reduce-scatter behind GEMMs (TE userbuffers)
FP8=${FP8:-0}                           # 1 = FP8 hybrid recipe
DEBUG_LOGGING=${DEBUG_LOGGING:-0}       # 1 = params-norm / num-zeros / straggler logging (costs ~1-2%)

echo "TOTAL_GPUS=${TOTAL_GPUS} GBS=${GBS} MBS=${MBS} TP=${TP} PP=${PP} SAVE_INTERVAL=${SAVE_INTERVAL}"
echo "ATTN_BACKEND=${ATTN_BACKEND} TP_COMM_OVERLAP=${TP_COMM_OVERLAP} FP8=${FP8} DEBUG_LOGGING=${DEBUG_LOGGING}"

MEGATRON_ARGS=(
  --attention-backend "$ATTN_BACKEND"
  --distributed-timeout-minutes 230
  --use-mcore-models
  --no-mmap-bin-files
  --sequence-parallel
  --untie-embeddings-and-output-weights
  --disable-bias-linear
  --init-method-std 0.014
  --position-embedding-type rope
  --rotary-base 1000000
  --rotary-percent 1.0
  --squared-relu
  --group-query-attention
  --kv-channels 128
  --normalization RMSNorm
  --attention-dropout 0.0
  --hidden-dropout 0.0
  --exit-duration-in-mins "$EXIT_DURATION_MINS"
  --train-iters "$TRAIN_ITERS"
  --lr-decay-iters 1830030
  --lr 6e-4
  --min-lr 6e-6
  --weight-decay 0.1
  --clip-grad 1.0
  --lr-decay-style cosine
  --lr-warmup-iters 5
  --eval-iters 1
  --eval-interval 50
  --log-interval 10
  --tokenizer-type NullTokenizer
  --vocab-size 131072
  --mock-data
  --num-workers 1
  --no-create-attention-mask-in-dataloader
  --log-progress
  --timing-log-option minmax
  --log-throughput
  --bf16
  --adam-beta1 0.9
  --adam-beta2 0.95
  --use-distributed-optimizer
  --overlap-grad-reduce
  --overlap-param-gather
  --manual-gc
  --check-weight-hash-across-dp-replicas-interval 20000
  --ckpt-fully-parallel-save
  --ckpt-fully-parallel-load
  --async-save
  --ckpt-assume-constant-structure
  --ckpt-format torch_dist
  --num-layers 79
  --hidden-size 8192
  --ffn-hidden-size 32768
  --num-attention-heads 64
  --seq-length 8192
  --max-position-embeddings 8192
  --num-query-groups 8
  --tensor-model-parallel-size "$TP"
  --pipeline-model-parallel-size "$PP"
  --micro-batch-size "$MBS"
  --global-batch-size "$GBS"
  --save "$CHECKPOINT_DIR"
  --data-cache-path "$DATA_CACHE_DIR"
  --save-interval "$SAVE_INTERVAL"
  --save-retain-interval "$SAVE_RETAIN_INTERVAL"
  --tensorboard-dir "$TENSORBOARD_DIR"
)

if [ "$TP_COMM_OVERLAP" = "1" ]; then
  MEGATRON_ARGS+=(--tp-comm-overlap)
fi
if [ "$FP8" = "1" ]; then
  MEGATRON_ARGS+=(--fp8-format hybrid --fp8-amax-history-len 1024 --fp8-amax-compute-algo max --fp8-param-gather)
fi
if [ "$DEBUG_LOGGING" = "1" ]; then
  MEGATRON_ARGS+=(--log-params-norm --log-num-zeros-in-grad
                  --log-straggler --disable-straggler-on-startup --straggler-minmax-count 16)
fi

# Resume only if a real checkpoint exists (core_v0.15 errors on --load of an empty dir).
if [ -f "${CHECKPOINT_DIR}/latest_checkpointed_iteration.txt" ]; then
  echo "Resuming from ${CHECKPOINT_DIR} at iteration $(cat ${CHECKPOINT_DIR}/latest_checkpointed_iteration.txt)"
  MEGATRON_ARGS+=(--load "$CHECKPOINT_DIR")
else
  echo "No checkpoint in ${CHECKPOINT_DIR}; starting from scratch"
fi

# ---------------- Launch ----------------
srun "${CONTAINER_ARGS[@]}" \
  torchrun \
    --nnodes="$SLURM_NNODES" \
    --nproc-per-node="$GPUS_PER_NODE" \
    --rdzv-backend=c10d \
    --rdzv-endpoint="${MASTER_ADDR}:${MASTER_PORT}" \
    --rdzv-id="$SLURM_JOB_ID" \
    "${MEGATRON_PATH}/pretrain_gpt.py" "${MEGATRON_ARGS[@]}"
```

A few lines in the script need explaining:

- **`--container-writable`.** TIR's Pyxis configuration mounts the container root filesystem read-only. Triton and Transformer Engine compile kernels into `~/.triton` at import time, so without this flag every rank dies before training starts.
- **`make` instead of `pip install -e .`.** The Megatron helper `.so` is the only thing that needs building. `make -C megatron/core/datasets` writes it into the source tree on `/pfs`; `PYTHONPATH` handles imports. This also avoids pip trying to write to `/root/.local`.
- **The conditional `--load`.** Megatron `core_v0.15.2` raises `is not a MLM or NeMo sharded checkpoint` if `--load` points at an empty directory. Passing it only when `latest_checkpointed_iteration.txt` exists makes the same script work for both the first run and every resume.

## Step 3: Pin NCCL to the 400G InfiniBand rails

Each TIR H200 node exposes 12 Mellanox devices, and only 8 of them are the InfiniBand compute rails. `ibv_devinfo` inside the container shows the split:

| Device | Adapter | Rate | Link layer | Role |
|---|---|---|---|---|
| `mlx5_0`, `mlx5_3`, `mlx5_4`, `mlx5_5`, `mlx5_6`, `mlx5_9`, `mlx5_10`, `mlx5_11` | ConnectX-7 (MT4129) | 400 Gb/s | InfiniBand | GPU compute rails, one per GPU |
| `mlx5_1`, `mlx5_2` | ConnectX-7 (MT4129) | 200 Gb/s | Ethernet | Storage / frontend |
| `mlx5_7`, `mlx5_8` | BlueField-3 (MT41692) | 200 Gb/s | Ethernet | Management |

Left to its own device selection, NCCL may pick up the Ethernet ports as RoCE and mix them into the rings. Pin it to the eight IB rails with an exact-match list:

```bash
export NCCL_IB_HCA==mlx5_0:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_9:1,mlx5_10:1,mlx5_11:1
export NCCL_SOCKET_IFNAME=eth0     # bootstrap / out-of-band only
```

The leading `=` after `NCCL_IB_HCA=` matters. Without it, NCCL treats each entry as a prefix: an entry of `mlx5_1` would also select `mlx5_10` and `mlx5_11`. With `=`, every name is an exact match, so the list selects exactly these eight ports, and `:1` pins port 1 on each card.

Run the first job with `NCCL_DEBUG=INFO` and look for one line per rank. This is what a correct setup prints on TIR:

```text
NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_3:1/IB/SHARP [2]mlx5_4:1/IB/SHARP [3]mlx5_5:1/IB/SHARP
         [4]mlx5_6:1/IB/SHARP [5]mlx5_9:1/IB/SHARP [6]mlx5_10:1/IB/SHARP [7]mlx5_11:1/IB/SHARP [RO]
Channel 04/0 : 12[4] -> 4[4] [receive] via NET/IBext_v10/4/GDRDMA
Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
```

Eight entries, all `/IB`, none `/RoCE`. Cross-node traffic goes over GDRDMA (GPUDirect RDMA, each GPU on its own rail), intra-node traffic over NVLink `P2P/CUMEM`, and the HPC-X plugin reports SHARP support. If you see `NET/Socket` instead, the InfiniBand devices are not visible inside the container. Device numbering can differ between nodes, so check `ibv_devinfo -l` on every node before trusting the list.

## Step 4: Checkpoint Megatron-LM every hour inside a 24-hour Slurm job

Megatron-LM has no wall-clock checkpoint option; `--save-interval` counts iterations. Getting an hourly cadence inside a 24-hour Slurm job means setting the save interval, the retain interval and the exit time together.

**Calibrate the interval from the measured step time.** After 30 to 50 iterations, read `elapsed time per iteration (ms)` from the log and set `SAVE_INTERVAL = floor(3600 / seconds_per_iteration)`. We calibrated on an early 25.5 s per iteration (3600 / 25.5 ≈ 141) and used 140; the run then settled at 24.5 s, so saves landed every 57 to 58 minutes rather than exactly 60. That is close enough for an hourly policy. Recalibrate whenever you change batch size, micro batch or precision: at the untuned 7.1 s per iteration and batch 16, the same hour was about 500 iterations.

**Keep the retain interval a multiple of the save interval.** `--save-retain-interval` deletes every checkpoint except the latest and those at multiples of its value. Megatron requires the retain interval to be a multiple of the save interval and stops at startup otherwise. The script derives it as `SAVE_INTERVAL * 6 = 840`, so one checkpoint every 6 hours survives the run: at about 3,300 iterations a day, that is iterations 840, 1,680 and 2,520, plus the latest.

**Exit cleanly before Slurm's time limit.** `--exit-duration-in-mins 1410` makes Megatron write a final checkpoint and exit 23 h 30 m after training starts, about half an hour before the 24-hour `--time`. Slurm's default `KillWait` is 30 seconds, far too short for a checkpoint of this size, so relying on SIGTERM at the limit would lose up to an hour of work. About 3,300 iterations fit in the window at 25.5 s each.

**Resuming is then a resubmit of the same script.** It finds `latest_checkpointed_iteration.txt`, adds `--load`, and continues from that iteration.

**Budget the storage before the first run.** A 56B model with the distributed optimizer's fp32 state writes on the order of 750 to 800 GB per checkpoint. `--async-save` hides most of the pause, but each write still has to finish within the hour, which needs about 225 MB/s of sustained write bandwidth. Three retained checkpoints, the latest, and briefly the one being replaced come to about 3 to 4 TB over a 24-hour job. TIR's [Parallel File System](https://www.e2enetworks.com/parallel-file-storage) handles this comfortably; a single NFS server generally does not.

## Step 5: Submit the job, monitor GPU utilisation and verify the first checkpoint

Submit from the login node. For the first run, keep `NCCL_DEBUG=INFO` on so the transport is in the log; drop it afterwards, it adds thousands of lines.

```bash
cd /pfs/nemotron-h
sbatch --export=ALL,NCCL_DEBUG=INFO nemotron_h_56b_2x8h200.sbatch
squeue -u $USER
tail -f /pfs/nemotron-h/logs/nemotron-h-56b-<JOBID>.out
```

Each phase of the log has a line that tells you it worked:

| Phase | Look for | Means |
|---|---|---|
| Prep step | `git clone` output, then `helpers_cpp*.so` found | Megatron-LM is on `/pfs` and built (first run only) |
| Startup | `world size: 16`, `data-parallel size: 2`, `... tensor-model-parallel size: 8` | Parallelism is what you asked for |
| Network | `NET/IB : Using [0]mlx5_0:1/IB ...` with 8 entries | Step 3 worked |
| Training | `iteration 20/ 100000 ... elapsed time per iteration (ms): 25425.5 ... TFLOP/s/GPU: 469.6 ... lm loss: 8.34` | Steps are running at the expected speed |

The TIR dashboard's Jobs tab mirrors `squeue`. Once training starts, the Monitoring tab should show average GPU utilisation near 100% on both nodes; the Nodes tab has the per-GPU view. If both nodes stay at 0% several minutes after the job starts, the rendezvous has probably not completed; see the last row of the troubleshooting table.

After about an hour, confirm the first checkpoint landed:

```bash
cat /pfs/nemotron-h/checkpoints/latest_checkpointed_iteration.txt   # 140
du -sh /pfs/nemotron-h/checkpoints/iter_0000140
grep "successfully saved checkpoint" /pfs/nemotron-h/logs/nemotron-h-56b-<JOBID>.out
```

With `--async-save`, the `iter_*` directory appears first and fills in over the next few minutes; the `successfully saved checkpoint` line marks the write finishing. To test resume, `scancel` the job after that line and resubmit: the log should show `/pfs/nemotron-h/checkpoints at iteration 140`. When the 24-hour job ends at 23 h 30 m, the same resubmit continues the run from its final checkpoint; uncomment `--dependency=singleton` in the header to queue the next day's job in advance.

## Results: 489 TFLOP/s per GPU, 49% MFU

The tuned configuration settles at 24.5 s per iteration and 487 to 489 TFLOP/s per GPU on 16 H200s, which is 49.2 to 49.4% MFU against the H200's 989 TFLOP/s dense BF16 peak. The first working run, before tuning, did 420 TFLOP/s (42.4%). The tuned run differs from it in global batch size, TP communication overlap and per-step diagnostics. The tuned column below is the steady state of the 24-hour run (iterations 400 to 2100); the first-run column is the average of iterations 30 to 60 of that job:

| Metric | First run | Tuned run |
|---|---|---|
| Global batch (micro batch 1) | 16 | 64 |
| TP communication overlap | off | on (`--tp-comm-overlap`) |
| Per-step diagnostics (params norm, num zeros, straggler) | on | off |
| Iteration time | 7.1 s | 24.5 s |
| Throughput (Megatron-reported) | 420 TFLOP/s per GPU | 487–489 TFLOP/s per GPU |
| Tokens per second, cluster | 18.4k | 21.4k |
| MFU (model FLOPs incl. attention, vs 989 TFLOP/s) | 42.4% | 49.2–49.4% |
| MFU (6ND convention, no attention) | 38.7% | 45.0% |
| GPU memory, peak allocated | not logged | 100.4 GB of 141 GB |

Common to both: Nemotron-H-56B, 56.5B parameters (79 layers, hidden 8192, FFN 32768, GQA 64/8), sequence 8192, BF16, TP=8, PP=1, DP=2, cross-node traffic over 8× 400G IB with GPUDirect RDMA.

Megatron's throughput per GPU is model FLOPs only, with no activation recompute in this config, so it is directly comparable to peak. A first-principles check agrees: about 364 GFLOP per token (6 × 55.4B for the non-embedding and output layers, plus about 32 GFLOP of causal attention at sequence 8192) × 524,288 tokens per step ÷ 24.5 s ÷ 16 GPUs ≈ 487 TFLOP/s.

We changed all three settings together, so what follows is our reading of where the 7 points came from, not a measured breakdown. `--tp-comm-overlap` hides the per-layer all-gather and reduce-scatter across the 8 TP ranks behind the GEMMs. A global batch of 64 gives each DP rank 32 microbatches per step instead of 8, so the optimizer step and gradient reduction are amortised 4×. Dropping `--log-params-norm`, `--log-num-zeros-in-grad` and `--log-straggler` removes an extra pass over the parameters and gradients every step. Step time grew 3.5× for 4× the tokens, which is where the fixed per-step cost shows up. The comparison also overstates the tuning gain somewhat: the first-run figure comes from its first few minutes, while the tuned figure is from after the two-hour ramp, and the tuned run's own early iterations were around 470 TFLOP/s.

Micro batch 2 is the first thing to try next. It doubles the GEMM M dimension and, from the measured 100 GB peak at micro batch 1, should land near 120 GB per GPU (add `--use-precision-aware-optimizer` if it does not fit). FlashAttention-3, which the container does not ship, is usually faster than the FlashAttention-2 kernels on Hopper, and cuDNN fused attention (`ATTN_BACKEND=fused`) is worth a comparison too. TP=4 with PP=2 halves the participants in every TP collective at the cost of a small pipeline bubble. FP8 (`FP8=1`) should raise throughput further, but we have not measured it on this setup, and its MFU is quoted against the FP8 peak, so it is not comparable to these BF16 numbers. Most of these are a single `sbatch --export` setting for the script above; test them one at a time over 80 iterations and compare the iteration 20 to 30 throughput.

### The 24-hour run

With the tuned defaults locked in, we submitted the 24-hour job at 10:35 on 29 September. The two charts below cover the first 18 hours of its log: throughput per GPU, and the checkpoint cadence.

![Line chart of Megatron-reported throughput per GPU across iterations 10 to 2,700 of the 24-hour run. Throughput starts near 396 TFLOP/s, jumps to about 470 within the first few iterations, ramps over roughly two hours to the high 480s, then holds between 487 and 489 TFLOP/s with a peak of 489.7, dipping by a few TFLOP/s at each hourly checkpoint. At iteration 2160 it steps down about 4% to roughly 466 TFLOP/s and stays there through iteration 2,700, ending at 468.7. A dashed reference line marks the untuned first run at 420 TFLOP/s with global batch 16 and no TP overlap.](https://strapi-main-website.objectstore.e2enetworks.net/training2_d41ebb94a3.png)

*Figure 2. Megatron-reported throughput per GPU. Source: Slurm job log, iterations 10–2,700, the first 18 hours of the 24-hour run, 29–30 Sep 2026.*

After a two-hour ramp from 469 to the high 480s, throughput sat between 487 and 489 TFLOP/s per GPU for 17 hours. Some of the hourly checkpoints show up as a one-sample dip of up to about 8 TFLOP/s. From iteration 2160 the run stepped down to about 466 TFLOP/s and stayed there. Nothing in the job configuration changed at that point, so the likely causes are outside the training loop: GPU clock or power throttling on one node, or a degraded InfiniBand link. The filesystem is an unlikely cause: the run reads synthetic data, and each checkpoint pauses training for only about 2.6 s. The TIR Monitoring tab records per-node GPU power and temperature, and the Nodes tab shows DCGM metrics and XID errors, so those are the first places to check, along with `nvidia-smi -q -d PERFORMANCE` on both nodes.

![Bar chart of how long training paused at each of the 19 checkpoint saves, from 11:35 on 29 September to 05:05 on 30 September. The first save stalls training for 46.6 seconds because PyTorch compiles the checkpoint extension on the fly. Every subsequent save stalls for about 2.6 seconds, with a mean of 2.58 seconds. Three bars are highlighted as retained under the 840-iteration retain interval, at roughly 16:26, 22:12 and 04:05, about six hours apart; the remaining 16 saves are deleted automatically after the next save.](https://strapi-main-website.objectstore.e2enetworks.net/training3_970b7b8c03.png)

*Figure 3. Training pause at each checkpoint save, and which saves were retained. Source: save-checkpoint timer and retention messages in the Slurm job log, 19 saves from 11:35 on 29 Sep to 05:05 on 30 Sep.*

Saves landed every 57 to 60 minutes (140 iterations at 24.4 to 25.6 s each). With `--async-save`, the training loop pauses only long enough to hand the sharded state to the background writer: about 2.6 s per save, roughly 0.1% of the hour. The exception was the first save, which took 46.6 s because PyTorch compiled the checkpoint extension on the fly; setting `TORCH_CUDA_ARCH_LIST=9.0` in the script should shorten that one-time compile. Retention matched the arithmetic: after each save, the previous one was deleted unless its iteration was a multiple of 840, so iterations 840, 1680 and 2520 survived while the rest were removed within the hour.

At TP=8, the same arithmetic as the TP=4 estimate gives roughly 80 to 85 GB per GPU for weights, gradients and optimizer state. Megatron reported 81.6 GB allocated and a 100.4 GB peak per GPU (102.3 GB reserved) out of 141 GB, with activations accounting for about 19 GB of the peak.

## Troubleshooting: the Megatron-LM and Pyxis errors we hit

Our first jobs failed within two minutes of starting, mostly because of how Pyxis sets up the container. All the fixes are already in the script; the table shows what each failure looks like in the log and how to fix it.

| Symptom in the log | Cause | Fix |
|---|---|---|
| `OSError: [Errno 30] Read-only file system: '/root/.triton'` on every rank at `import transformer_engine` | TIR's Pyxis configuration mounts the container rootfs read-only; Triton compiles its CUDA utils into `~/.triton` | `--container-writable` on `srun`, plus `TRITON_CACHE_DIR=/tmp/...` so JIT caches stay node-local |
| `ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/root/.local'` in the prep step | `pip install -e .` (the usual way to make Megatron importable) writes to the user site-packages | Build only the helper: `make -C megatron/core/datasets`; imports come from `PYTHONPATH` |
| `ValueError: /pfs/nemotron-h/checkpoints is not a MLM or NeMo sharded checkpoint!` from `has_modelopt_state` | `--load` pointed at an empty directory; `core_v0.15.2` checks for a ModelOpt state before building the model | Add `--load` only when `latest_checkpointed_iteration.txt` exists |
| Job fails instantly with no `.out` file | Slurm could not open `--output` because the log directory did not exist | `mkdir -p /pfs/nemotron-h/logs` before the first `sbatch` |
| `NET/Socket` instead of `NET/IB` in NCCL output | `/dev/infiniband` not visible in the container, or `NCCL_IB_HCA` names do not exist on that node | Check `ls /dev/infiniband` and `ibv_devinfo -l` inside the container on both nodes |
| Hang at rendezvous, then a c10d timeout | Workers cannot reach `MASTER_ADDR:29500` | Check that the `MASTER_ADDR` hostname resolves on both nodes and that port 29500 is reachable between them. After rendezvous, `NCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` must match the pod network interface |

You can ignore `Failed to create unified memory mempool` and `TORCH_NCCL_AVOID_RECORD_STREAMS is the default now`; both appear on every rank and do not affect the run. A grad norm spike in the first hundred iterations (we saw 91 at iteration 60 after values around 1) is expected with only a 5-iteration warm-up and is clipped by `--clip-grad 1.0`; raise `--lr-warmup-iters` to a few hundred for a production run.

## From benchmark to real training

This job trains Nemotron-H-56B from scratch on Megatron-LM's built-in synthetic token stream, so the numbers above measure the cluster, not a data pipeline. To run it on your own data, replace `--mock-data` and `--tokenizer-type NullTokenizer` with a `--data-path` blend of tokenized Megatron `.bin`/`.idx` files and the matching tokenizer. Raise `--lr-warmup-iters` from 5 to a few hundred and pick a `--global-batch-size` that suits your token budget. Set `--train-iters` and `--lr-decay-iters` for the run you actually intend, then recalibrate `SAVE_INTERVAL` from the new step time. Throughput will be within a few percent of what you see here, since the data pipeline is not the bottleneck at this scale.

To scale out, set `--nodes=4` on a 4-node TIR cluster. That gives DP=4 with the same TP=8, and the c10d rendezvous and the IB pinning carry over unchanged. Recalibrate `SAVE_INTERVAL`, because the step time roughly halves and 140 iterations would then take about 30 minutes. With the global batch still at 64, each DP rank gets 16 microbatches instead of 32, so consider raising `--global-batch-size`.  Test a resume from a 2-node checkpoint before relying on it.

## Run your own training job on TIR

The cluster in this post is available to rent by the hour, with the same H200 nodes, 400G InfiniBand fabric and Parallel File System. [Explore training and fine-tuning on TIR](https://www.e2enetworks.com/training-fine-tuning) to size a cluster for your own run, or talk to our team about multi-node capacity.

## Frequently Asked Questions

### What MFU can you expect from Megatron-LM on NVIDIA H200 GPUs?

On 16 H200 GPUs across 2 nodes we measured 487 to 489 TFLOP/s per GPU, which is 49.2 to 49.4% MFU against the H200's 989 TFLOP/s dense BF16 peak. That is for a 56.5B-parameter dense Transformer at sequence length 8192 in BF16, with TP=8, PP=1, DP=2 and no activation recompute. Before tuning, the same job ran at 420 TFLOP/s (42.4% MFU); the difference came from raising the global batch from 16 to 64, enabling `--tp-comm-overlap`, and turning off per-step diagnostic logging.

### How much does it cost to run a 24-hour training job on 16 H200 GPUs?

At E2E Networks' on-demand rate of ₹502.55 per GPU-hour, 16 H200 GPUs cost ₹8,040.80 per hour, so a 24-hour job comes to about ₹1,92,979.20 (384 GPU-hours). In USD that is \$2031.36 at \$5.29 per GPU-hour. At the throughput measured here — 21.4k tokens per second — that run processes roughly 1.8 billion tokens, or about ₹80,000 per billion tokens. Committed-use and spot rates are lower; see the [GPU pricing page](https://www.e2enetworks.com/pricing) for current rates.

### Why does NCCL_IB_HCA need a leading equals sign?

Without the leading `=`, NCCL treats each name in the list as a prefix, so an entry of `mlx5_1` also matches `mlx5_10` and `mlx5_11`. Writing `NCCL_IB_HCA==mlx5_0:1,mlx5_3:1,...` makes every name an exact match, and the `:1` suffix pins port 1 on each card. On a node that exposes both InfiniBand and Ethernet adapters, getting this wrong lets NCCL pull Ethernet ports into the rings as RoCE and silently costs you throughput.

### How do you make Megatron-LM checkpoint every hour?

Megatron-LM has no wall-clock checkpoint option — `--save-interval` counts iterations — so you calibrate it from the measured step time. Run 30 to 50 iterations, read `elapsed time per iteration (ms)` from the log, and set `--save-interval` to `floor(3600 / seconds_per_iteration)`. At 25.5 s per iteration that is 140. Recalibrate after any change to batch size, micro batch, precision or node count, because the step time moves with all of them.

### Why does a Megatron-LM container need --container-writable on a Pyxis Slurm cluster?

Pyxis mounts the container root filesystem read-only by default, and Transformer Engine makes Triton compile CUDA kernels into `~/.triton` at import time. Without `--container-writable` every rank dies with `OSError: [Errno 30] Read-only file system: '/root/.triton'` before training starts. Pair the flag with `TRITON_CACHE_DIR=/tmp/triton-cache` so the JIT caches stay on node-local disk instead of shared storage.

### Why use TP=8 instead of TP=4 for a 56B model on H200?

With pipeline parallelism off (PP=1), TP=4 leaves roughly 126 GB per GPU for bf16 weights, fp32 gradients and the sharded optimizer state before activations, which does not fit on a 141 GB H200 at sequence length 8192. TP=8 brings that down to roughly 80 to 85 GB; the measured peak was 100.4 GB including activations. TP=8 also keeps every tensor-parallel collective inside one node on NVLink, so only the DP=2 gradient all-reduce crosses InfiniBand.

### How do you resume a Slurm training job from a Megatron-LM checkpoint?

Resubmit the same sbatch script. The script checks for `latest_checkpointed_iteration.txt` in the checkpoint directory and adds `--load` only when it exists, so the first run starts from scratch and every later run continues from the last save. Passing `--load` unconditionally fails on an empty directory in `core_v0.15.2` with `is not a MLM or NeMo sharded checkpoint`. Add `--dependency=singleton` to the `#SBATCH` header to queue the next day's job in advance.

### How much storage does a 56B-parameter checkpoint need?

About 750 to 800 GB per checkpoint for a 56B model with the distributed optimizer's fp32 state. Writing one inside an hourly cadence needs roughly 225 MB/s of sustained write bandwidth. Over a 24-hour job that retains one checkpoint every 6 hours, budget 3 to 4 TB. That is comfortable for a parallel file system and generally beyond a single NFS server.

## Sources

- [TIR Slurm cluster concepts, E2E Networks docs](https://docs.e2enetworks.com/docs/tir/SlurmCluster/getting-started/slurm-cluster-concepts/)
- [TIR Training Cluster: running training jobs, E2E Networks docs](https://docs.e2enetworks.com/docs/tir/TrainingCluster/deployments/)
- [Megatron-LM core_v0.15.2, NVIDIA](https://github.com/NVIDIA/Megatron-LM)
- [NVIDIA PyTorch container 25.08, NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)
- [NCCL environment variables, NVIDIA docs](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html)
- [Pyxis usage, NVIDIA on GitHub](https://github.com/NVIDIA/pyxis/wiki/Usage)