Add ROCm nightly + PyTorch profiling tutorial

MLExamples/PyTorch_Profiling/README_ROCM_NIGHTLY_TESTING.md

@@ -0,0 +1,129 @@
+# Testing a ROCm Nightly Build with PyTorch (CIFAR-100 train + profile)
+
+This guide walks you through testing a **nightly ROCm build** together with
+PyTorch on an AMD GPU. You will:
+
+1. Build a self-contained Python virtual environment with nightly ROCm, PyTorch,
+   and the ROCm profilers.
+2. Create a `setup_rocm.sh` script that activates that environment.
+3. Run the CIFAR-100 training workload through each profiling tool using the
+   provided SLURM scripts, and check that everything works end to end.
+
+The workload itself is the same `train_cifar_100.py` used throughout this
+directory — a small vision model trained on CIFAR-100. It is intentionally
+short so that a nightly build can be validated quickly.
+
+> This guide targets an AMD MI300A GPU (`gfx942`) on a SLURM cluster. For a
+> different GPU, change the architecture (`device-gfx942` / `--arch`) and the
+> SLURM `--partition` in the scripts accordingly.
+
+---
+
+## Step 1 — Build the virtual environment
+
+Follow [`ROCM_PYTORCH_PIP_VENV_SETUP.md`](./ROCM_PYTORCH_PIP_VENV_SETUP.md) to
+create the `rocm-pytorch-pip` venv. In short, it:
+
+- creates a venv at `~/venvs/rocm-pytorch-pip`,
+- installs nightly ROCm + PyTorch + profilers from the multi-arch nightly index,
+- installs `transformers` (required by the training script),
+- runs `rocm-sdk init` to extract the development headers and device code.
+
+To test a **specific nightly**, set the ROCm version pin in that guide, e.g.:
+
+```bash
+ROCM_VERSION=7.15.0a20260721
+```
+
+Change this value to the nightly date you want to validate.
+
+## Step 2 — Verify `setup_rocm.sh`
+
+The SLURM scripts in each sub-directory activate the environment by sourcing
+`../setup_rocm.sh` — i.e. the `setup_rocm.sh` shipped in **this**
+(`PyTorch_Profiling/`) directory. It is already provided; just verify (and edit
+if needed) that its `VENV` points at the venv you built in Step 1. Its full
+contents, and a GPU-node sanity check, are covered in steps 5-7 of
+[`ROCM_PYTORCH_PIP_VENV_SETUP.md`](./ROCM_PYTORCH_PIP_VENV_SETUP.md).
+
+Once verified, a quick check that the nightly build runs GPU kernels through
+PyTorch:
+
+```bash
+source setup_rocm.sh
+srun -n1 --gpus=1 python3 -c "import torch; print('torch', torch.__version__); \
+x = torch.ones(4, device='cuda:0'); print('device ok:', (x+1).sum().item())"
+```
+
+If you see `device ok:`, you are ready to profile.
+
+## Step 3 — Run the SLURM scripts
+
+Each sub-directory contains a single-process SLURM script that sources
+`../setup_rocm.sh`, pre-downloads the dataset if needed, and runs the workload
+under one tool. All of them use a single GPU and a short run
+(`--batch-size 32 --max-steps 5`) so a nightly can be checked quickly.
+
+**Submit each script from its own directory** (the scripts use
+`SLURM_SUBMIT_DIR` to locate themselves):
+
+| Tool | Directory | Script | What it produces |
+|------|-----------|--------|------------------|
+| None (baseline) | `no-profiling/` | `slurm_single_process_noprofile.sh` | Plain training run — confirms the workload runs without a profiler. |
+| ROCm Compute Profiler (profile) | `rocm-compute-profiler/` | `slurm_single_process_profile.sh` | Hardware-counter profile under `workloads/`. |
+| ROCm Compute Profiler (analyze) | `rocm-compute-profiler/` | `slurm_single_process_analyze.sh` | Analysis report from the profile above. |
+| RocProfiler (kernels) | `rocprofv3/` | `slurm_single_process_kernels.sh` | Kernel stats + trace CSVs under `single_process/`. |
+| RocProfiler (traces) | `rocprofv3/` | `slurm_single_process_traces.sh` | System timeline trace (`.pftrace`) under `single_process/`. |
+| ROCm Systems Profiler | `rocm-systems-profiler/` | `slurm_single_process.sh` | Sampling profile + trace under `rocprofsys-python3-output/`. |
+| Roofline Extractor | `roofline-extractor/` | `slurm_single_process.sh` | Per-kernel roofline analysis + interactive HTML plot under `output/`. |
+
+Example (baseline sanity check first, then a profiler):
+
+```bash
+cd no-profiling
+sbatch slurm_single_process_noprofile.sh
+
+cd ../rocprofv3
+sbatch slurm_single_process_kernels.sh
+```
+
+Check job status and output:
+
+```bash
+squeue --me
+# stdout/stderr land in the submit directory as <name>_<jobid>.out / .err
+```
+
+### Analyzing the results
+
+> **Note — running `rocprof-compute analyze`:** run it only **after** its
+> profile job has finished (the counter database must exist). Because it needs
+> `numpy==1.26.4` (vs the shared venv's `numpy>=2.0`), the analysis script uses
+> its own isolated venv (`~/venvs/rocprof-compute-analyze`) and never touches
+> the shared venv.
+
+- **ROCm Compute Profiler:** submit the companion analysis job from
+  `rocm-compute-profiler/` (it locates the workload and runs the analysis for you):
+
+```bash
+cd rocm-compute-profiler
+sbatch slurm_single_process_analyze.sh
+```
+
+  Or run it by hand: `rocprof-compute analyze -p rocm-compute-profiler/workloads/cifar_100_single_proc/<subdir>`
+- **RocProfiler:** open the CSVs (kernels) or load the `.pftrace` in
+  [ui.perfetto.dev](https://ui.perfetto.dev) (traces).
+- **ROCm Systems Profiler:** load the `perfetto-trace-*.proto` from
+  `rocprofsys-python3-output/<timestamp>/` in [ui.perfetto.dev](https://ui.perfetto.dev).
+- **Roofline Extractor:** open the generated `.html` in `roofline-extractor/output/`.
+
+---
+
+## Notes
+
+- If a job fails to start, check the SLURM `--partition` and time limits in the
+  script headers match your cluster.
+- The scripts derive a per-job rendezvous port from the SLURM job ID, so
+  multiple jobs can share a node without port collisions.
+- To validate a different nightly, rebuild the venv (Step 1) with a new
+  `ROCM_VERSION` and re-run the scripts.

MLExamples/PyTorch_Profiling/ROCM_PYTORCH_PIP_VENV_SETUP.md

@@ -0,0 +1,123 @@
+# Creating the `rocm-pytorch-pip` venv (nightly ROCm + PyTorch, MI300A / gfx942)
+
+This guide walks you through building a Python virtual environment with ROCm,
+PyTorch, and the ROCm profiling tools, so you can train and profile a model on
+an AMD GPU. Follow the steps in order.
+
+Everything installs from pip into one self-contained venv, so PyTorch and the
+profilers use the same ROCm. Each step below is a single command block you can
+copy and run.
+
+> This guide targets an AMD MI300A GPU (`gfx942`). If you have a different GPU,
+> change `device-gfx942` to your architecture.
+
+---
+
+## 1. Create and activate the venv
+
+First choose where the venv should live. Set `VENV_BASE` to the directory that
+will hold the `venvs` folder (defaults to your home directory). Keep this shell
+open for the remaining steps, which reuse the variable.
+
+```bash
+VENV_BASE=~
+mkdir -p "${VENV_BASE}/venvs"
+python -m venv "${VENV_BASE}/venvs/rocm-pytorch-pip"
+source "${VENV_BASE}/venvs/rocm-pytorch-pip/bin/activate"
+```
+
+## 2. Install ROCm + PyTorch from the nightly multi-arch index
+
+```bash
+# Pin the nightly ROCm version once and reuse it everywhere below.
+ROCM_VERSION=7.15.0a20260721
+
+pip install --index-url https://rocm.nightlies.amd.com/whl-multi-arch/ \
+    "rocm[profiler,devel,libraries,device-gfx942]==${ROCM_VERSION}" \
+    "torch[device-gfx942]" \
+    "torchvision[device-gfx942]"
+```
+
+The `rocm[...]` extras pull in the pieces this workflow needs:
+- `profiler`  — the ROCm profilers: `rocprof-compute`, `rocprofv3`, and
+  `rocprof-sys` (bundled `_rocm_profiler`)
+- `devel`     — development package (headers/device code, extracted in step 4)
+- `libraries` — math libraries (hipBLAS, rocBLAS, ...)
+- `device-gfx942` — the GPU-arch kernels for MI300A
+
+## 3. Install `transformers` (required by the training script)
+
+```bash
+pip install transformers
+```
+
+The training script builds its models with `transformers`, so this package is
+required.
+
+## 4. Extract development headers and device code
+
+```bash
+"${VENV_BASE}/venvs/rocm-pytorch-pip/bin/rocm-sdk" init
+```
+
+`rocm-sdk init` unpacks the `devel` payload (headers, LLVM device bitcode) into
+`_rocm_sdk_devel/` inside the venv — the device bitcode HIP needs to run kernels,
+and the paths `setup_rocm.sh` points at next.
+
+---
+
+## 5. Verify `setup_rocm.sh`
+
+The repo already ships `setup_rocm.sh` in `MLExamples/PyTorch_Profiling/` (the
+SLURM scripts source it as `../setup_rocm.sh`). It activates the venv and points
+the ROCm environment at the extracted `_rocm_sdk_devel` tree. It defaults
+`VENV_BASE` to your home directory; if you used a different `VENV_BASE` in
+step 1, export it before sourcing (or edit the default here). Its contents are:
+
+```bash
+#!/usr/bin/env bash
+# Source this to activate the ROCm venv and set ROCm env vars:
+#   source setup_rocm.sh
+VENV_BASE="${VENV_BASE:-$HOME}"
+VENV="$VENV_BASE/venvs/rocm-pytorch-pip"
+source "$VENV/bin/activate"
+DEVEL="$(python3 -c 'import site; print(site.getsitepackages()[0])')/_rocm_sdk_devel"
+export ROCM_PATH="$DEVEL"
+export HIP_PATH="$DEVEL"
+export HIP_DEVICE_LIB_PATH="$DEVEL/lib/llvm/amdgcn/bitcode"
+export PATH="$DEVEL/bin:$PATH"
+export LD_LIBRARY_PATH="$DEVEL/lib:$DEVEL/lib/rocm_sysdeps/lib:$LD_LIBRARY_PATH"
+echo "ROCm venv active: $VENV"
+```
+
+## 6. Re-source to pick up the ROCm env vars
+
+If the venv is already active from step 1, deactivate and source the script so
+the `ROCM_PATH` / `LD_LIBRARY_PATH` exports take effect (run from
+`MLExamples/PyTorch_Profiling/`):
+
+```bash
+deactivate
+source setup_rocm.sh
+```
+
+---
+
+## 7. Verify (on a GPU node)
+
+```bash
+source setup_rocm.sh
+srun -n1 --gpus=1 python3 -c "import torch; print('torch', torch.__version__); \
+x = torch.ones(4, device='cuda:0'); print('device ok:', (x+1).sum().item())"
+```
+
+Expected output resembles:
+
+```
+ROCm venv active: /.../rocm-pytorch-pip
+torch 2.12.0+rocm7.15.0a20260721
+device ok: 8.0
+```
+
+If you see `device ok:`, the GPU is working and your environment is ready to use.
+From now on, just run `source setup_rocm.sh` in any new shell to activate it.

MLExamples/PyTorch_Profiling/no-profiling/slurm_single_process_noprofile.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+#SBATCH --job-name=rpc-single-noprofile
+#SBATCH --nodes=1
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=8
+#SBATCH --gpus=1
+#SBATCH --time=00:30:00
+#SBATCH --output=rpc_single_process_noprofile_%j.out
+#SBATCH --error=rpc_single_process_noprofile_%j.err
+
+# ---------------------------------------------------------------------------
+# Run the single-process CIFAR-100 workload directly with python3 (no profiler).
+# Sources ../setup_rocm.sh to activate the ROCm PyTorch venv.
+# ---------------------------------------------------------------------------
+
+set -e
+
+# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from
+# MLExamples/PyTorch_Profiling/no-profiling/.
+if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then
+    SCRIPT_DIR="${SLURM_SUBMIT_DIR}"
+else
+    SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")"
+fi
+PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")"
+echo "SCRIPT_DIR=${SCRIPT_DIR}"
+echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}"
+
+# ../setup_rocm.sh activates the ROCm PyTorch venv and exports ROCm env vars.
+source ${PROFILER_TOP_DIR}/setup_rocm.sh
+
+# Distributed bootstrap variables expected by train_cifar_100.py (single rank).
+export NPROCS=1
+export MASTER_ADDR=${MASTER_ADDR:-$(hostname)}
+# Derive a per-job port so concurrent jobs don't collide.
+export MASTER_PORT=${MASTER_PORT:-$((20000 + SLURM_JOB_ID % 20000))}
+
+# Make sure the dataset is present.
+if [ ! -d ${PROFILER_TOP_DIR}/data/cifar-100-python ]; then
+    python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \
+        --data-path ${PROFILER_TOP_DIR}/data --download-only
+fi
+
+cd ${SCRIPT_DIR}
+
+# Run the workload directly (no profiler).
+echo
+echo "==================================================================="
+echo "Running (no profiler): python3 train_cifar_100.py"
+echo "==================================================================="
+srun -n 1 --gpus=1 --cpus-per-task=8 \
+    python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \
+        --batch-size 32 --max-steps 5 \
+        --data-path ${PROFILER_TOP_DIR}/data
+
+echo
+echo "==================================================================="
+echo "Workload run complete."
+echo "==================================================================="