diff --git a/MLExamples/PyTorch_Profiling/README_ROCM_NIGHTLY_TESTING.md b/MLExamples/PyTorch_Profiling/README_ROCM_NIGHTLY_TESTING.md new file mode 100644 index 00000000..e9a66f75 --- /dev/null +++ b/MLExamples/PyTorch_Profiling/README_ROCM_NIGHTLY_TESTING.md @@ -0,0 +1,129 @@ +# Testing a ROCm Nightly Build with PyTorch (CIFAR-100 train + profile) + +This guide walks you through testing a **nightly ROCm build** together with +PyTorch on an AMD GPU. You will: + +1. Build a self-contained Python virtual environment with nightly ROCm, PyTorch, + and the ROCm profilers. +2. Create a `setup_rocm.sh` script that activates that environment. +3. Run the CIFAR-100 training workload through each profiling tool using the + provided SLURM scripts, and check that everything works end to end. + +The workload itself is the same `train_cifar_100.py` used throughout this +directory — a small vision model trained on CIFAR-100. It is intentionally +short so that a nightly build can be validated quickly. + +> This guide targets an AMD MI300A GPU (`gfx942`) on a SLURM cluster. For a +> different GPU, change the architecture (`device-gfx942` / `--arch`) and the +> SLURM `--partition` in the scripts accordingly. + +--- + +## Step 1 — Build the virtual environment + +Follow [`ROCM_PYTORCH_PIP_VENV_SETUP.md`](./ROCM_PYTORCH_PIP_VENV_SETUP.md) to +create the `rocm-pytorch-pip` venv. In short, it: + +- creates a venv at `~/venvs/rocm-pytorch-pip`, +- installs nightly ROCm + PyTorch + profilers from the multi-arch nightly index, +- installs `transformers` (required by the training script), +- runs `rocm-sdk init` to extract the development headers and device code. + +To test a **specific nightly**, set the ROCm version pin in that guide, e.g.: + +```bash +ROCM_VERSION=7.15.0a20260721 +``` + +Change this value to the nightly date you want to validate. + +## Step 2 — Verify `setup_rocm.sh` + +The SLURM scripts in each sub-directory activate the environment by sourcing +`../setup_rocm.sh` — i.e. the `setup_rocm.sh` shipped in **this** +(`PyTorch_Profiling/`) directory. It is already provided; just verify (and edit +if needed) that its `VENV` points at the venv you built in Step 1. Its full +contents, and a GPU-node sanity check, are covered in steps 5-7 of +[`ROCM_PYTORCH_PIP_VENV_SETUP.md`](./ROCM_PYTORCH_PIP_VENV_SETUP.md). + +Once verified, a quick check that the nightly build runs GPU kernels through +PyTorch: + +```bash +source setup_rocm.sh +srun -n1 --gpus=1 python3 -c "import torch; print('torch', torch.__version__); \ +x = torch.ones(4, device='cuda:0'); print('device ok:', (x+1).sum().item())" +``` + +If you see `device ok:`, you are ready to profile. + +## Step 3 — Run the SLURM scripts + +Each sub-directory contains a single-process SLURM script that sources +`../setup_rocm.sh`, pre-downloads the dataset if needed, and runs the workload +under one tool. All of them use a single GPU and a short run +(`--batch-size 32 --max-steps 5`) so a nightly can be checked quickly. + +**Submit each script from its own directory** (the scripts use +`SLURM_SUBMIT_DIR` to locate themselves): + +| Tool | Directory | Script | What it produces | +|------|-----------|--------|------------------| +| None (baseline) | `no-profiling/` | `slurm_single_process_noprofile.sh` | Plain training run — confirms the workload runs without a profiler. | +| ROCm Compute Profiler (profile) | `rocm-compute-profiler/` | `slurm_single_process_profile.sh` | Hardware-counter profile under `workloads/`. | +| ROCm Compute Profiler (analyze) | `rocm-compute-profiler/` | `slurm_single_process_analyze.sh` | Analysis report from the profile above. | +| RocProfiler (kernels) | `rocprofv3/` | `slurm_single_process_kernels.sh` | Kernel stats + trace CSVs under `single_process/`. | +| RocProfiler (traces) | `rocprofv3/` | `slurm_single_process_traces.sh` | System timeline trace (`.pftrace`) under `single_process/`. | +| ROCm Systems Profiler | `rocm-systems-profiler/` | `slurm_single_process.sh` | Sampling profile + trace under `rocprofsys-python3-output/`. | +| Roofline Extractor | `roofline-extractor/` | `slurm_single_process.sh` | Per-kernel roofline analysis + interactive HTML plot under `output/`. | + +Example (baseline sanity check first, then a profiler): + +```bash +cd no-profiling +sbatch slurm_single_process_noprofile.sh + +cd ../rocprofv3 +sbatch slurm_single_process_kernels.sh +``` + +Check job status and output: + +```bash +squeue --me +# stdout/stderr land in the submit directory as _.out / .err +``` + +### Analyzing the results + +> **Note — running `rocprof-compute analyze`:** run it only **after** its +> profile job has finished (the counter database must exist). Because it needs +> `numpy==1.26.4` (vs the shared venv's `numpy>=2.0`), the analysis script uses +> its own isolated venv (`~/venvs/rocprof-compute-analyze`) and never touches +> the shared venv. + +- **ROCm Compute Profiler:** submit the companion analysis job from + `rocm-compute-profiler/` (it locates the workload and runs the analysis for you): + +```bash +cd rocm-compute-profiler +sbatch slurm_single_process_analyze.sh +``` + + Or run it by hand: `rocprof-compute analyze -p rocm-compute-profiler/workloads/cifar_100_single_proc/` +- **RocProfiler:** open the CSVs (kernels) or load the `.pftrace` in + [ui.perfetto.dev](https://ui.perfetto.dev) (traces). +- **ROCm Systems Profiler:** load the `perfetto-trace-*.proto` from + `rocprofsys-python3-output//` in [ui.perfetto.dev](https://ui.perfetto.dev). +- **Roofline Extractor:** open the generated `.html` in `roofline-extractor/output/`. + +--- + +## Notes + +- If a job fails to start, check the SLURM `--partition` and time limits in the + script headers match your cluster. +- The scripts derive a per-job rendezvous port from the SLURM job ID, so + multiple jobs can share a node without port collisions. +- To validate a different nightly, rebuild the venv (Step 1) with a new + `ROCM_VERSION` and re-run the scripts. diff --git a/MLExamples/PyTorch_Profiling/ROCM_PYTORCH_PIP_VENV_SETUP.md b/MLExamples/PyTorch_Profiling/ROCM_PYTORCH_PIP_VENV_SETUP.md new file mode 100644 index 00000000..1d023d59 --- /dev/null +++ b/MLExamples/PyTorch_Profiling/ROCM_PYTORCH_PIP_VENV_SETUP.md @@ -0,0 +1,123 @@ +# Creating the `rocm-pytorch-pip` venv (nightly ROCm + PyTorch, MI300A / gfx942) + +This guide walks you through building a Python virtual environment with ROCm, +PyTorch, and the ROCm profiling tools, so you can train and profile a model on +an AMD GPU. Follow the steps in order. + +Everything installs from pip into one self-contained venv, so PyTorch and the +profilers use the same ROCm. Each step below is a single command block you can +copy and run. + +> This guide targets an AMD MI300A GPU (`gfx942`). If you have a different GPU, +> change `device-gfx942` to your architecture. + +--- + +## 1. Create and activate the venv + +First choose where the venv should live. Set `VENV_BASE` to the directory that +will hold the `venvs` folder (defaults to your home directory). Keep this shell +open for the remaining steps, which reuse the variable. + +```bash +VENV_BASE=~ +mkdir -p "${VENV_BASE}/venvs" +python -m venv "${VENV_BASE}/venvs/rocm-pytorch-pip" +source "${VENV_BASE}/venvs/rocm-pytorch-pip/bin/activate" +``` + +## 2. Install ROCm + PyTorch from the nightly multi-arch index + +```bash +# Pin the nightly ROCm version once and reuse it everywhere below. +ROCM_VERSION=7.15.0a20260721 + +pip install --index-url https://rocm.nightlies.amd.com/whl-multi-arch/ \ + "rocm[profiler,devel,libraries,device-gfx942]==${ROCM_VERSION}" \ + "torch[device-gfx942]" \ + "torchvision[device-gfx942]" +``` + +The `rocm[...]` extras pull in the pieces this workflow needs: +- `profiler` — the ROCm profilers: `rocprof-compute`, `rocprofv3`, and + `rocprof-sys` (bundled `_rocm_profiler`) +- `devel` — development package (headers/device code, extracted in step 4) +- `libraries` — math libraries (hipBLAS, rocBLAS, ...) +- `device-gfx942` — the GPU-arch kernels for MI300A + +## 3. Install `transformers` (required by the training script) + +```bash +pip install transformers +``` + +The training script builds its models with `transformers`, so this package is +required. + +## 4. Extract development headers and device code + +```bash +"${VENV_BASE}/venvs/rocm-pytorch-pip/bin/rocm-sdk" init +``` + +`rocm-sdk init` unpacks the `devel` payload (headers, LLVM device bitcode) into +`_rocm_sdk_devel/` inside the venv — the device bitcode HIP needs to run kernels, +and the paths `setup_rocm.sh` points at next. + +--- + +## 5. Verify `setup_rocm.sh` + +The repo already ships `setup_rocm.sh` in `MLExamples/PyTorch_Profiling/` (the +SLURM scripts source it as `../setup_rocm.sh`). It activates the venv and points +the ROCm environment at the extracted `_rocm_sdk_devel` tree. It defaults +`VENV_BASE` to your home directory; if you used a different `VENV_BASE` in +step 1, export it before sourcing (or edit the default here). Its contents are: + +```bash +#!/usr/bin/env bash +# Source this to activate the ROCm venv and set ROCm env vars: +# source setup_rocm.sh +VENV_BASE="${VENV_BASE:-$HOME}" +VENV="$VENV_BASE/venvs/rocm-pytorch-pip" +source "$VENV/bin/activate" +DEVEL="$(python3 -c 'import site; print(site.getsitepackages()[0])')/_rocm_sdk_devel" +export ROCM_PATH="$DEVEL" +export HIP_PATH="$DEVEL" +export HIP_DEVICE_LIB_PATH="$DEVEL/lib/llvm/amdgcn/bitcode" +export PATH="$DEVEL/bin:$PATH" +export LD_LIBRARY_PATH="$DEVEL/lib:$DEVEL/lib/rocm_sysdeps/lib:$LD_LIBRARY_PATH" +echo "ROCm venv active: $VENV" +``` + +## 6. Re-source to pick up the ROCm env vars + +If the venv is already active from step 1, deactivate and source the script so +the `ROCM_PATH` / `LD_LIBRARY_PATH` exports take effect (run from +`MLExamples/PyTorch_Profiling/`): + +```bash +deactivate +source setup_rocm.sh +``` + +--- + +## 7. Verify (on a GPU node) + +```bash +source setup_rocm.sh +srun -n1 --gpus=1 python3 -c "import torch; print('torch', torch.__version__); \ +x = torch.ones(4, device='cuda:0'); print('device ok:', (x+1).sum().item())" +``` + +Expected output resembles: + +``` +ROCm venv active: /.../rocm-pytorch-pip +torch 2.12.0+rocm7.15.0a20260721 +device ok: 8.0 +``` + +If you see `device ok:`, the GPU is working and your environment is ready to use. +From now on, just run `source setup_rocm.sh` in any new shell to activate it. diff --git a/MLExamples/PyTorch_Profiling/no-profiling/slurm_single_process_noprofile.sh b/MLExamples/PyTorch_Profiling/no-profiling/slurm_single_process_noprofile.sh new file mode 100755 index 00000000..11fef41b --- /dev/null +++ b/MLExamples/PyTorch_Profiling/no-profiling/slurm_single_process_noprofile.sh @@ -0,0 +1,59 @@ +#!/bin/bash +#SBATCH --job-name=rpc-single-noprofile +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --gpus=1 +#SBATCH --time=00:30:00 +#SBATCH --output=rpc_single_process_noprofile_%j.out +#SBATCH --error=rpc_single_process_noprofile_%j.err + +# --------------------------------------------------------------------------- +# Run the single-process CIFAR-100 workload directly with python3 (no profiler). +# Sources ../setup_rocm.sh to activate the ROCm PyTorch venv. +# --------------------------------------------------------------------------- + +set -e + +# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from +# MLExamples/PyTorch_Profiling/no-profiling/. +if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then + SCRIPT_DIR="${SLURM_SUBMIT_DIR}" +else + SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")" +fi +PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")" +echo "SCRIPT_DIR=${SCRIPT_DIR}" +echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}" + +# ../setup_rocm.sh activates the ROCm PyTorch venv and exports ROCm env vars. +source ${PROFILER_TOP_DIR}/setup_rocm.sh + +# Distributed bootstrap variables expected by train_cifar_100.py (single rank). +export NPROCS=1 +export MASTER_ADDR=${MASTER_ADDR:-$(hostname)} +# Derive a per-job port so concurrent jobs don't collide. +export MASTER_PORT=${MASTER_PORT:-$((20000 + SLURM_JOB_ID % 20000))} + +# Make sure the dataset is present. +if [ ! -d ${PROFILER_TOP_DIR}/data/cifar-100-python ]; then + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --data-path ${PROFILER_TOP_DIR}/data --download-only +fi + +cd ${SCRIPT_DIR} + +# Run the workload directly (no profiler). +echo +echo "===================================================================" +echo "Running (no profiler): python3 train_cifar_100.py" +echo "===================================================================" +srun -n 1 --gpus=1 --cpus-per-task=8 \ + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --batch-size 32 --max-steps 5 \ + --data-path ${PROFILER_TOP_DIR}/data + +echo +echo "===================================================================" +echo "Workload run complete." +echo "===================================================================" diff --git a/MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_analyze.sh b/MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_analyze.sh new file mode 100755 index 00000000..e587d8ae --- /dev/null +++ b/MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_analyze.sh @@ -0,0 +1,139 @@ +#!/bin/bash +#SBATCH --job-name=rpc-single-analyze +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --time=00:15:00 +#SBATCH --output=rpc_single_process_analyze_%j.out +#SBATCH --error=rpc_single_process_analyze_%j.err + +# --------------------------------------------------------------------------- +# Analyze the profiled workload with rocprof-compute analyze. +# +# rocprof-compute analyze needs numpy 1.26.x, which conflicts with the shared +# venv's numpy 2.x. So we use a separate venv holding only rocprof-compute's +# pinned requirements, while reusing the tool bundled in the shared venv (the +# shared venv is never modified). CPU-only; run after the profile job finishes. +# --------------------------------------------------------------------------- + +set -e + +# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from +# MLExamples/PyTorch_Profiling/rocm-compute-profiler/. +if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then + SCRIPT_DIR="${SLURM_SUBMIT_DIR}" +else + SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")" +fi +PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")" +echo "SCRIPT_DIR=${SCRIPT_DIR}" +echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}" + +# Locate the shared ROCm venv WITHOUT activating it; we only need the ROCm +# install inside it (bundled rocprof-compute launcher + ROCm libraries). +MAIN_VENV="${HOME}/venvs/rocm-pytorch-pip" +if [[ ! -x "${MAIN_VENV}/bin/python3" ]]; then + echo "ERROR: shared ROCm venv not found at ${MAIN_VENV}" >&2 + exit 1 +fi +# site-packages of the shared venv (avoids hard-coding the python3.x version). +MAIN_SP="$("${MAIN_VENV}/bin/python3" -c 'import site; print(site.getsitepackages()[0])')" +ROCM_PROFILER="${MAIN_SP}/_rocm_profiler" +ROCM_CORE="${MAIN_SP}/_rocm_sdk_core" +ROCM_DEVEL="${MAIN_SP}/_rocm_sdk_devel" +COMPUTE_LIBEXEC="${ROCM_PROFILER}/libexec/rocprofiler-compute" +REQ_FILE="${COMPUTE_LIBEXEC}/requirements.txt" +RPC="${ROCM_PROFILER}/bin/rocprof-compute" + +for p in "${RPC}" "${REQ_FILE}" "${ROCM_CORE}" "${ROCM_DEVEL}"; do + if [[ ! -e "${p}" ]]; then + echo "ERROR: expected ROCm component not found: ${p}" >&2 + exit 1 + fi +done + +# Create/refresh the dedicated analysis venv with rocprof-compute's pinned +# requirements (numpy 1.26.x). Reinstalled only when requirements.txt changes. +ANALYZE_VENV="${HOME}/venvs/rocprof-compute-analyze" +if [[ ! -x "${ANALYZE_VENV}/bin/python3" ]]; then + echo "Creating analysis venv at ${ANALYZE_VENV}" + # A venv stays isolated from its creator's site-packages, so using the + # shared venv's python3 keeps this independent while avoiding system python3. + "${MAIN_VENV}/bin/python3" -m venv "${ANALYZE_VENV}" +fi +source "${ANALYZE_VENV}/bin/activate" + +STAMP="${ANALYZE_VENV}/.rocprof_compute_reqs_installed" +if [[ ! -f "${STAMP}" || "${REQ_FILE}" -nt "${STAMP}" ]]; then + echo "Installing rocprof-compute requirements into ${ANALYZE_VENV}" + python3 -m pip install --upgrade pip + python3 -m pip install -r "${REQ_FILE}" + cp "${REQ_FILE}" "${STAMP}" +else + echo "Analysis venv already satisfies ${REQ_FILE}" +fi + +# Point the isolated venv at the ROCm install in the shared venv (bundled +# rocprof-compute + libs). ROCm bin goes LAST on PATH so it can't shadow the +# analysis venv's python3, keeping numpy 1.26.x in effect. +export PYTHONPATH="${COMPUTE_LIBEXEC}:${PYTHONPATH}" +export LD_LIBRARY_PATH="${ROCM_CORE}/lib:${ROCM_CORE}/lib/rocm_sysdeps/lib:${ROCM_DEVEL}/lib:${ROCM_DEVEL}/lib/rocm_sysdeps/lib:${LD_LIBRARY_PATH}" +export ROCM_PATH="${ROCM_DEVEL}" +export HIP_PATH="${ROCM_DEVEL}" +export PATH="${PATH}:${ROCM_DEVEL}/bin" + +echo "Analysis python : $(which python3) (numpy $(python3 -c 'import numpy; print(numpy.__version__)'))" +"${RPC}" --version + +# Locate the workload written by slurm_single_process_profile.sh (must match +# WORKLOAD_NAME / WORK_ROOT there). +WORKLOAD_NAME=cifar_100_single_proc +WORK_ROOT=${SCRIPT_DIR}/workloads +WORK_DIR=${WORK_ROOT}/${WORKLOAD_NAME} + +if [[ ! -d "${WORK_DIR}" ]]; then + echo "ERROR: workload directory not found: ${WORK_DIR}" >&2 + echo "Run slurm_single_process_profile.sh first." >&2 + exit 1 +fi + +# The profile is written into a single subdirectory named after the architecture +# (e.g. `MI300A_*`) or numerically (`0`, `1`, ...). Pick the first one. +ARCH_DIR=$(find ${WORK_DIR} -mindepth 1 -maxdepth 1 -type d | sort | head -1) +if [[ -z "${ARCH_DIR}" ]]; then + echo "ERROR: no workload subdirectory found under ${WORK_DIR}" >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# Analyze. +# --------------------------------------------------------------------------- +# Optional: list per-kernel stats and dispatch IDs. Uncomment only if you want +# to narrow the analysis to a specific kernel/dispatch (see the NOTE at the +# bottom of this file). +# echo +# echo "===================================================================" +# echo "rocprof-compute analyze --list-stats -p ${ARCH_DIR}" +# echo "===================================================================" +# STATS_FILE=${SCRIPT_DIR}/stats_${SLURM_JOB_ID}.txt +# "${RPC}" analyze --list-stats -p ${ARCH_DIR} >& ${STATS_FILE} +# echo "Stats and dispatch IDs written to ${STATS_FILE}" + +echo +echo "===================================================================" +echo "rocprof-compute analyze -p ${ARCH_DIR}" +echo "===================================================================" +ANALYSIS_FILE=${SCRIPT_DIR}/analysis_${SLURM_JOB_ID}.txt +"${RPC}" analyze -p ${ARCH_DIR} > ${ANALYSIS_FILE} 2>&1 +echo "Analysis written to ${ANALYSIS_FILE}" + +echo +echo "-------------------------------------------------------------------" +echo "NOTE: The analysis above aggregates all kernels. To focus on a" +echo "specific kernel, uncomment the --list-stats block above to list the" +echo "kernel names and dispatch IDs, then re-run rocprof-compute analyze" +echo "narrowing to that kernel with --kernel , or to a specific" +echo "invocation with --dispatch (you may pass either, or both):" +echo " ${RPC} analyze -p ${ARCH_DIR} --kernel " +echo " ${RPC} analyze -p ${ARCH_DIR} --dispatch " +echo "-------------------------------------------------------------------" diff --git a/MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_profile.sh b/MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_profile.sh new file mode 100755 index 00000000..be3974ed --- /dev/null +++ b/MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_profile.sh @@ -0,0 +1,84 @@ +#!/bin/bash +#SBATCH --job-name=rpc-single-profile +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --gpus=1 +#SBATCH --time=02:00:00 +#SBATCH --output=rpc_single_process_profile_%j.out +#SBATCH --error=rpc_single_process_profile_%j.err + +# --------------------------------------------------------------------------- +# Profile the single-process CIFAR-100 workload with rocprof-compute profile. +# Sources ../setup_rocm.sh to activate the ROCm PyTorch venv. +# --------------------------------------------------------------------------- + +set -e + +# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from +# MLExamples/PyTorch_Profiling/rocm-compute-profiler/. +if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then + SCRIPT_DIR="${SLURM_SUBMIT_DIR}" +else + SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")" +fi +PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")" +echo "SCRIPT_DIR=${SCRIPT_DIR}" +echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}" + +# ../setup_rocm.sh activates the ROCm PyTorch venv and exports ROCm env vars. +source ${PROFILER_TOP_DIR}/setup_rocm.sh +rocprof-compute --version + +# Distributed bootstrap vars for train_cifar_100.py (single rank). Derive a +# per-job port so concurrent jobs don't collide. +export NPROCS=1 +export MASTER_ADDR=${MASTER_ADDR:-$(hostname)} +export MASTER_PORT=${MASTER_PORT:-$((20000 + SLURM_JOB_ID % 20000))} + +# Pre-download the dataset so the profiled step doesn't spend time on it. +if [ ! -d ${PROFILER_TOP_DIR}/data/cifar-100-python ]; then + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --data-path ${PROFILER_TOP_DIR}/data --download-only +fi + +WORKLOAD_NAME=cifar_100_single_proc +WORK_ROOT=${SCRIPT_DIR}/workloads +WORK_DIR=${WORK_ROOT}/${WORKLOAD_NAME} + +mkdir -p ${WORK_ROOT} +rm -rf ${WORK_DIR} +cd ${SCRIPT_DIR} + +# Profile. Notes: +# * --no-roof skips roofline capture (not needed here). +# * rocprof-compute replays the app multiple times to cover all counters, so +# keep the workload short. +echo +echo "===================================================================" +echo "rocprof-compute profile -- python3 train_cifar_100.py" +echo "===================================================================" +srun -n 1 --gpus=1 --cpus-per-task=8 \ + rocprof-compute profile \ + --name ${WORKLOAD_NAME} \ + --no-roof \ + -- \ + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --batch-size 32 --max-steps 5 \ + --data-path ${PROFILER_TOP_DIR}/data + +# The profile subdir is named by arch (e.g. MI300A_*) or numerically; pick the first. +ARCH_DIR=$(find ${WORK_DIR} -mindepth 1 -maxdepth 1 -type d | sort | head -1) +if [[ -z "${ARCH_DIR}" ]]; then + echo "ERROR: no workload subdirectory found under ${WORK_DIR}" >&2 + exit 1 +fi + +echo +echo "===================================================================" +echo "Profile complete." +echo "Workload directory: ${ARCH_DIR}" +echo +echo "Analyze with, e.g.:" +echo " rocprof-compute analyze -p ${ARCH_DIR}" +echo "===================================================================" diff --git a/MLExamples/PyTorch_Profiling/rocm-systems-profiler/slurm_single_process.sh b/MLExamples/PyTorch_Profiling/rocm-systems-profiler/slurm_single_process.sh new file mode 100755 index 00000000..c4e73749 --- /dev/null +++ b/MLExamples/PyTorch_Profiling/rocm-systems-profiler/slurm_single_process.sh @@ -0,0 +1,65 @@ +#!/bin/bash +#SBATCH --job-name=rps-single-run +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --gpus=1 +#SBATCH --time=00:30:00 +#SBATCH --output=rps_single_process_%j.out +#SBATCH --error=rps_single_process_%j.err + +# --------------------------------------------------------------------------- +# Profile the single-process CIFAR-100 workload with the ROCm Systems Profiler +# (rocprof-sys-run). Sources ../setup_rocm.sh to activate the ROCm venv. +# --------------------------------------------------------------------------- + +set -e + +# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from +# MLExamples/PyTorch_Profiling/rocm-systems-profiler/. +if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then + SCRIPT_DIR="${SLURM_SUBMIT_DIR}" +else + SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")" +fi +PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")" +echo "SCRIPT_DIR=${SCRIPT_DIR}" +echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}" + +# ../setup_rocm.sh activates the ROCm PyTorch venv and exports ROCm env vars. +source ${PROFILER_TOP_DIR}/setup_rocm.sh +rocprof-sys-run --version + +# Distributed bootstrap variables expected by train_cifar_100.py (single rank). +export NPROCS=1 +export MASTER_ADDR=${MASTER_ADDR:-$(hostname)} +# Derive a per-job port so concurrent jobs don't collide. +export MASTER_PORT=${MASTER_PORT:-$((20000 + SLURM_JOB_ID % 20000))} + +# Make sure the dataset is present before the profiled step. +if [ ! -d ${PROFILER_TOP_DIR}/data/cifar-100-python ]; then + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --data-path ${PROFILER_TOP_DIR}/data --download-only +fi + +cd ${SCRIPT_DIR} + +# Remove any stale output directories from previous runs. +rm -rf ${SCRIPT_DIR}/rocprofsys-python3-output + +# Profile with rocprof-sys-run (profiling + tracing). +echo +echo "===================================================================" +echo "rocprof-sys-run --profile --trace -- python3 train_cifar_100.py" +echo "===================================================================" +srun -n 1 --gpus=1 --cpus-per-task=8 \ + rocprof-sys-run --profile --trace -- \ + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --batch-size 32 --max-steps 5 \ + --data-path ${PROFILER_TOP_DIR}/data + +echo +echo "===================================================================" +echo "Profile complete." +echo "Output directory: ${SCRIPT_DIR}/rocprofsys-python3-output" +echo "===================================================================" diff --git a/MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_kernels.sh b/MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_kernels.sh new file mode 100755 index 00000000..e4cb75a4 --- /dev/null +++ b/MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_kernels.sh @@ -0,0 +1,66 @@ +#!/bin/bash +#SBATCH --job-name=rpv3-single-kernels +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --gpus=1 +#SBATCH --time=00:30:00 +#SBATCH --output=rpv3_single_process_kernels_%j.out +#SBATCH --error=rpv3_single_process_kernels_%j.err + +# --------------------------------------------------------------------------- +# Profile the single-process CIFAR-100 workload with rocprofv3 kernel tracing +# (--stats --kernel-trace). Sources ../setup_rocm.sh to activate the ROCm venv. +# --------------------------------------------------------------------------- + +set -e + +# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from +# MLExamples/PyTorch_Profiling/rocprofv3/. +if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then + SCRIPT_DIR="${SLURM_SUBMIT_DIR}" +else + SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")" +fi +PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")" +echo "SCRIPT_DIR=${SCRIPT_DIR}" +echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}" + +# ../setup_rocm.sh activates the ROCm PyTorch venv and exports ROCm env vars. +source ${PROFILER_TOP_DIR}/setup_rocm.sh +rocprofv3 --version + +# Distributed bootstrap variables expected by train_cifar_100.py (single rank). +export NPROCS=1 +export MASTER_ADDR=${MASTER_ADDR:-$(hostname)} +# Derive a per-job port so concurrent jobs don't collide. +export MASTER_PORT=${MASTER_PORT:-$((20000 + SLURM_JOB_ID % 20000))} + +# Make sure the dataset is present before the profiled step. +if [ ! -d ${PROFILER_TOP_DIR}/data/cifar-100-python ]; then + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --data-path ${PROFILER_TOP_DIR}/data --download-only +fi + +OUT_DIR=${SCRIPT_DIR}/single_process +rm -rf ${OUT_DIR} +cd ${SCRIPT_DIR} + +# Profile kernels with rocprofv3. +echo +echo "===================================================================" +echo "rocprofv3 --stats --kernel-trace -- python3 train_cifar_100.py" +echo "===================================================================" +srun -n 1 --gpus=1 --cpus-per-task=8 \ + rocprofv3 --stats --kernel-trace --output-format csv \ + --output-directory ${OUT_DIR} --output-file kernels \ + -- \ + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --batch-size 32 --max-steps 5 \ + --data-path ${PROFILER_TOP_DIR}/data + +echo +echo "===================================================================" +echo "Profile complete." +echo "Output directory: ${OUT_DIR}" +echo "===================================================================" diff --git a/MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_traces.sh b/MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_traces.sh new file mode 100755 index 00000000..910890ab --- /dev/null +++ b/MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_traces.sh @@ -0,0 +1,66 @@ +#!/bin/bash +#SBATCH --job-name=rpv3-single-traces +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --gpus=1 +#SBATCH --time=00:30:00 +#SBATCH --output=rpv3_single_process_traces_%j.out +#SBATCH --error=rpv3_single_process_traces_%j.err + +# --------------------------------------------------------------------------- +# Collect GPU timeline traces for the single-process CIFAR-100 workload with +# rocprofv3 (--sys-trace). Sources ../setup_rocm.sh to activate the ROCm venv. +# --------------------------------------------------------------------------- + +set -e + +# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from +# MLExamples/PyTorch_Profiling/rocprofv3/. +if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then + SCRIPT_DIR="${SLURM_SUBMIT_DIR}" +else + SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")" +fi +PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")" +echo "SCRIPT_DIR=${SCRIPT_DIR}" +echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}" + +# ../setup_rocm.sh activates the ROCm PyTorch venv and exports ROCm env vars. +source ${PROFILER_TOP_DIR}/setup_rocm.sh +rocprofv3 --version + +# Distributed bootstrap variables expected by train_cifar_100.py (single rank). +export NPROCS=1 +export MASTER_ADDR=${MASTER_ADDR:-$(hostname)} +# Derive a per-job port so concurrent jobs don't collide. +export MASTER_PORT=${MASTER_PORT:-$((20000 + SLURM_JOB_ID % 20000))} + +# Make sure the dataset is present before the profiled step. +if [ ! -d ${PROFILER_TOP_DIR}/data/cifar-100-python ]; then + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --data-path ${PROFILER_TOP_DIR}/data --download-only +fi + +OUT_DIR=${SCRIPT_DIR}/single_process +rm -rf ${OUT_DIR} +cd ${SCRIPT_DIR} + +# Collect GPU timeline traces with rocprofv3. +echo +echo "===================================================================" +echo "rocprofv3 --sys-trace -- python3 train_cifar_100.py" +echo "===================================================================" +srun -n 1 --gpus=1 --cpus-per-task=8 \ + rocprofv3 --sys-trace --output-format pftrace \ + --output-directory ${OUT_DIR} --output-file traces \ + -- \ + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --batch-size 32 --max-steps 5 \ + --data-path ${PROFILER_TOP_DIR}/data + +echo +echo "===================================================================" +echo "Trace collection complete." +echo "Output directory: ${OUT_DIR}" +echo "===================================================================" diff --git a/MLExamples/PyTorch_Profiling/roofline-extractor/slurm_single_process.sh b/MLExamples/PyTorch_Profiling/roofline-extractor/slurm_single_process.sh new file mode 100755 index 00000000..b98a1691 --- /dev/null +++ b/MLExamples/PyTorch_Profiling/roofline-extractor/slurm_single_process.sh @@ -0,0 +1,80 @@ +#!/bin/bash +#SBATCH --job-name=roofline-single +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --gpus=1 +#SBATCH --time=02:00:00 +#SBATCH --output=roofline_single_process_%j.out +#SBATCH --error=roofline_single_process_%j.err + +# --------------------------------------------------------------------------- +# Collect roofline plots for the single-process CIFAR-100 workload with +# AMD-HPC's rooflineExtractor (https://github.com/AMD-HPC/rooflineExtractor). +# Its profile_app.py runs rocprofv3 several times to collect counters + a kernel +# trace, then produces the per-kernel roofline analysis and an HTML plot. +# Sources ../setup_rocm.sh to activate the ROCm venv. +# --------------------------------------------------------------------------- + +set -e + +# Resolve this script's dir; under sbatch prefer SLURM_SUBMIT_DIR. Submit from +# MLExamples/PyTorch_Profiling/roofline-extractor/. +if [[ -n "${SLURM_SUBMIT_DIR}" ]]; then + SCRIPT_DIR="${SLURM_SUBMIT_DIR}" +else + SCRIPT_DIR="$(dirname "$(readlink -fm "$0")")" +fi +PROFILER_TOP_DIR="$(dirname "${SCRIPT_DIR}")" +echo "SCRIPT_DIR=${SCRIPT_DIR}" +echo "PROFILER_TOP_DIR=${PROFILER_TOP_DIR}" + +# ../setup_rocm.sh activates the ROCm PyTorch venv and exports ROCm env vars. +source ${PROFILER_TOP_DIR}/setup_rocm.sh +rocprofv3 --version + +# Fetch rooflineExtractor and install its Python dependencies (into the venv). +RE_DIR=${SCRIPT_DIR}/rooflineExtractor +if [ ! -d ${RE_DIR} ]; then + git clone https://github.com/AMD-HPC/rooflineExtractor.git ${RE_DIR} +fi +python3 -m pip install -r ${RE_DIR}/requirements.txt + +# Distributed bootstrap variables expected by train_cifar_100.py (single rank). +export NPROCS=1 +export MASTER_ADDR=${MASTER_ADDR:-$(hostname)} +# Derive a per-job port so concurrent jobs don't collide. +export MASTER_PORT=${MASTER_PORT:-$((20000 + SLURM_JOB_ID % 20000))} + +# Make sure the dataset is present before the profiled runs (profile_app.py runs +# the application several times, so pre-downloading avoids repeated downloads). +if [ ! -d ${PROFILER_TOP_DIR}/data/cifar-100-python ]; then + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --data-path ${PROFILER_TOP_DIR}/data --download-only +fi + +OUT_DIR=${SCRIPT_DIR}/output +rm -rf ${OUT_DIR} +cd ${SCRIPT_DIR} + +# Collect roofline data and generate plots. --arch MI300A selects the gfx942 +# counter set; profile_app.py runs rocprofv3 itself, so it is the target. +echo +echo "===================================================================" +echo "profile_app.py --arch MI300A -- python3 train_cifar_100.py" +echo "===================================================================" +srun -n 1 --gpus=1 --cpus-per-task=8 \ + python3 ${RE_DIR}/profile_app.py \ + --arch MI300A \ + -o ${OUT_DIR} \ + -- \ + python3 ${PROFILER_TOP_DIR}/train_cifar_100.py \ + --batch-size 32 --max-steps 5 \ + --data-path ${PROFILER_TOP_DIR}/data + +echo +echo "===================================================================" +echo "Roofline analysis complete." +echo "Output directory (counters, traces, plots, analysis): ${OUT_DIR}" +echo "Open the generated .html file for the interactive roofline plot." +echo "===================================================================" diff --git a/MLExamples/PyTorch_Profiling/setup_rocm.sh b/MLExamples/PyTorch_Profiling/setup_rocm.sh new file mode 100644 index 00000000..cb2998f5 --- /dev/null +++ b/MLExamples/PyTorch_Profiling/setup_rocm.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +# Source this to activate the ROCm venv and set ROCm env vars: +# source setup_rocm.sh +VENV_BASE="${VENV_BASE:-$HOME}" +VENV="${VENV_BASE}/venvs/rocm-pytorch-pip" +source "${VENV}/bin/activate" +# Derive site-packages from the active venv so this works regardless of the +# venv's Python minor version (e.g. python3.12 vs python3.13). +DEVEL="$(python3 -c 'import site; print(site.getsitepackages()[0])')/_rocm_sdk_devel" +export ROCM_PATH="$DEVEL" +export HIP_PATH="$DEVEL" +export HIP_DEVICE_LIB_PATH="${DEVEL}/lib/llvm/amdgcn/bitcode" +export PATH="${DEVEL}/bin:$PATH" +export LD_LIBRARY_PATH="${DEVEL}/lib:${DEVEL}/lib/rocm_sysdeps/lib:$LD_LIBRARY_PATH" +echo "ROCm venv active: $VENV" +