Add ROCm nightly + PyTorch profiling tutorial#173
Open
gsitaram wants to merge 6 commits into
Open
Conversation
… are not tested yet
There was a problem hiding this comment.
Pull request overview
Adds a ROCm-nightly + PyTorch validation/profiling mini-workflow under MLExamples/PyTorch_Profiling/, centered on a short CIFAR-100 training run and a set of single-process SLURM scripts to exercise multiple ROCm profiling tools.
Changes:
- Adds end-to-end documentation for building a pip-based ROCm+PyTorch venv and running the profiling workflow.
- Adds a
setup_rocm.shenvironment activation script and multiple single-process SLURM scripts (baseline + per-profiler runs). - Adds a dedicated
rocprof-compute analyzeSLURM script that provisions an isolated analysis venv to avoid numpy version conflicts.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| MLExamples/PyTorch_Profiling/setup_rocm.sh | Activates the ROCm PyTorch venv and exports ROCm-related environment variables. |
| MLExamples/PyTorch_Profiling/no-profiling/slurm_single_process_noprofile.sh | Baseline single-process CIFAR-100 run without profilers. |
| MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_kernels.sh | Single-process kernel stats/trace capture with rocprofv3. |
| MLExamples/PyTorch_Profiling/rocprofv3/slurm_single_process_traces.sh | Single-process timeline trace capture with rocprofv3 --sys-trace. |
| MLExamples/PyTorch_Profiling/rocm-systems-profiler/slurm_single_process.sh | Single-process Systems Profiler run via rocprof-sys-run. |
| MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_profile.sh | Single-process rocprof-compute profile capture workflow. |
| MLExamples/PyTorch_Profiling/rocm-compute-profiler/slurm_single_process_analyze.sh | CPU-only rocprof-compute analyze flow using an isolated venv for pinned deps. |
| MLExamples/PyTorch_Profiling/roofline-extractor/slurm_single_process.sh | Single-process rooflineExtractor workflow wrapper. |
| MLExamples/PyTorch_Profiling/ROCM_PYTORCH_PIP_VENV_SETUP.md | Step-by-step guide for creating the nightly ROCm+PyTorch pip venv. |
| MLExamples/PyTorch_Profiling/README_ROCM_NIGHTLY_TESTING.md | Top-level end-to-end guide tying setup + per-tool SLURM scripts together. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+4
to
+6
| VENV="${HOME}/venvs/rocm-pytorch-pip" | ||
| DEVEL="${VENV}/lib/python3.12/site-packages/_rocm_sdk_devel" | ||
| source "${VENV}/bin/activate" |
| Each sub-directory contains a single-process SLURM script that sources | ||
| `../setup_rocm.sh`, pre-downloads the dataset if needed, and runs the workload | ||
| under one tool. All of them use a single GPU and a short run | ||
| (`--batch-size 64 --max-steps 5`) so a nightly can be checked quickly. |
Comment on lines
+42
to
+45
| The SLURM scripts in each sub-directory activate the environment by sourcing | ||
| `../setup_rocm.sh` — i.e. a `setup_rocm.sh` located in **this** | ||
| (`PyTorch_Profiling/`) directory. Create it here (see step 5 of the venv guide | ||
| for the full contents), pointing at the venv you just built: |
Comment on lines
+80
to
+81
| Create `~/setup_rocm.sh` to activate the venv and point the ROCm environment at | ||
| the extracted `_rocm_sdk_devel` tree. Adjust `python3.12` to your Python version. |
Comment on lines
+12
to
+13
| # SLURM script: profile the single-process CIFAR-100 workload with the ROCm | ||
| # Systems Profiler (rocprof-sys-sample). |
| if [ ! -d ${RE_DIR} ]; then | ||
| git clone https://github.com/AMD-HPC/rooflineExtractor.git ${RE_DIR} | ||
| fi | ||
| pip install -r ${RE_DIR}/requirements.txt |
Comment on lines
+79
to
+81
| echo "Creating analysis venv at ${ANALYZE_VENV}" | ||
| /usr/bin/python3 -m venv "${ANALYZE_VENV}" | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a self-contained workflow for validating a nightly ROCm build with PyTorch by profiling a short CIFAR-100 training run across the ROCm profiling tools. All new files live under MLExamples/PyTorch_Profiling/.
Docs:
README_ROCM_NIGHTLY_TESTING.md(end-to-end guide) andROCM_PYTORCH_PIP_VENV_SETUP.md(pip venv setup), plussetup_rocm.shto activate the environment.Single-process SLURM scripts for each tool: baseline (
no-profiling/), ROCm Compute Profiler (rocm-compute-profiler/), rocprofv3 kernels + traces (rocprofv3/), ROCm Systems Profiler (rocm-systems-profiler/), and roofline extractor (roofline-extractor/).The rocprof-compute analyze step runs in its own isolated venv (
~/venvs/rocprof-compute-analyze) so its numpy==1.26.4 pin never disturbs the shared PyTorch venv (numpy>=2.0).