Skip to content

fix(installer): stop the ROCm package lookup from aborting the installer - #401

Merged
LukasWodka merged 6 commits into
developfrom
fix/gpu-amd-pipefail-hazard
Jul 27, 2026
Merged

fix(installer): stop the ROCm package lookup from aborting the installer#401
LukasWodka merged 6 commits into
developfrom
fix/gpu-amd-pipefail-hazard

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Summary

_find_package_name in scripts/lib/gpu-amd.sh ran the fetch, the match and head -1 as one pipeline and returned its status. scripts/install-k8s.sh sources this lib under set -euo pipefail, so that pipeline could take the installer down two different ways:

1. A failed fetch aborted the installer before the friendly error could report it. On a 404, timeout or proxy block the command substitution is non-zero, the caller's assignment inherits that status, and set -e aborts — before the [[ -z "$deb_name" ]] && error "No amdgpu-install .deb found at …" sitting on the very next line. The operator saw a silent abort mid-GPU-step instead of an actionable message. The RHEL major-version fallback (/rhel/9/ after /rhel/9.4/ misses) was unreachable for the same reason — the first empty result killed the script rather than falling through.

2. head -1 could SIGPIPE grep, failing a lookup that had actually succeeded. head -1 closes the pipe as soon as it has its line; if grep is still writing it takes SIGPIPE (141), and pipefail propagates that as a pipeline failure even though the filename was found. This only triggers once grep's output exceeds the pipe buffer, so it fails on large mirror indexes and passes everywhere else — the worst failure mode to leave in an installer.

The fix

Capture the fetch and the match separately, let neither fail the function, and take the first match with ${matches%%$'\n'*} so head leaves the pipeline entirely:

_find_package_name() {
  local dir_url="$1" ext="$2" index="" matches=""
  index="$(curl_secure -fsSL --max-time 30 "$dir_url")" || return 0
  matches="$(printf '%s\n' "$index" | grep -oE "amdgpu-install[^\"<>]*\.${ext}")" || return 0
  printf '%s\n' "${matches%%$'\n'*}"
}

Chosen over sprinkling || true, per the ticket discussion: || true on the original pipeline would have silenced the fetch failure but left the head/SIGPIPE interaction in place. The contract is unchanged — the filename on stdout, nothing at all when not found — so emptiness stays the single signal all three call sites already test for, and each one now reaches its own error message.

No set -euo pipefail added to the lib: scripts/lib/*.sh deliberately have none (the entrypoint sets it at install-k8s.sh:44).

Tests

Adds scripts/tests/gpu-amd.batsthe first coverage this lib has had. Seven cases: the three contract cases (finds the file, returns only the first of several, honours the requested extension), both hazards, and a caller-shaped regression that runs the real assignment under set -euo pipefail and asserts the error path is reached rather than the shell dying.

Mutation-checked. Reverting the function to its previous form fails exactly the two hazard tests (6 and 7) while the five contract tests keep passing — so the new tests detect these defects, and the contract tests independently confirm the happy path is unchanged.

The large-index test generates 20 000 matches so the SIGPIPE case is deterministic here, rather than depending on a real mirror's index size.

Verification

  • bats scripts/tests/gpu-amd.bats — 7/7 pass
  • bats scripts/tests/*.bats — 427 pass. Three failures (validate_config: valid config passes, un-stamped DEFAULT_REF fails closed, _extract_yaml_value: single-quoted with '' escape) are pre-existing: identical by name on the unmodified base branch, and local-environment artifacts on macOS bash 3.2 — the suite's own notes say Linux CI is the authority.
  • shellcheck --severity=error and --severity=warning clean; bash -n clean
  • bash scripts/check-style.sh — style + terminology clean
  • scripts/gen-manifest.sh re-run; scripts/manifest.sha256 committed (R8 supply-chain gate)

Base branch — please note

This is stacked on fix/1252-curl-secure-wrapper (#400), not on develop, because #400 rewrites the same line to use curl_secure. Targeting develop would have produced a guaranteed conflict on that line, and a careless resolution could silently drop the TLS floor #400 exists to add. Merge #400 first; GitHub will retarget this to develop automatically. The hazard itself predates #400 — it is present on develop today, just with plain curl.

Found while fixing backend#1252 in #400 and deliberately left out of that PR to keep it behaviour-neutral.


Note

Medium Risk
Touches the GPU install path in the k8s installer; behavior is intentionally narrowed to always exit 0 with empty output on failure, which is well covered by new tests but still affects production ROCm setup.

Overview
_find_package_name in scripts/lib/gpu-amd.sh no longer runs fetch, grep, and head -1 as one pipeline. The index fetch and regex match are captured in separate steps, each allowed to fail without exiting the function (|| return 0), so callers under set -euo pipefail still get empty stdout instead of a silent abort. The first match is taken with ${matches%%$'\n'*} instead of head -1, avoiding SIGPIPE/pipefail failures on large mirror listings while keeping the same contract (filename on stdout, empty when not found).

Adds scripts/tests/gpu-amd.bats with seven cases covering happy path, extension filtering, failed fetch, no match, large-index pipefail, and a caller-shaped set -euo pipefail regression. Updates scripts/manifest.sha256 for the changed lib.

Reviewed by Cursor Bugbot for commit aeb22fe. Bugbot is set up for automated code reviews on this repo. Configure here.

claude added 2 commits July 25, 2026 10:22
… (backend#1252)

`CURL_SECURE` was a bare constant every call site had to splice in by hand, so
call sites kept losing it: seven live `curl` invocations ran with no minimum TLS
version, including the POST in `verify_credentials()` that carries the client's
password. These installs run on customer-managed hosts and behind TLS-inspecting
proxies, which negotiate down to whatever the client permits — the reason this
repo adopted an explicit floor instead of trusting curl's defaults.

Add `curl_secure()` in `scripts/lib/common.sh` and route every fetch in
`scripts/lib/*.sh` through it (18 call sites). The wrapper always passes
`--tlsv1.2` and supplies default `--connect-timeout 30` / `--max-time 300`.
Defaults are injected before `"$@"`, so a call site that wants a tighter bound
still wins (curl honours the last occurrence), and a transfer that bounds itself
with `--speed-limit`/`--speed-time` gets no injected `--max-time` — a hard
deadline would fail a slow-but-healthy link on a large binary download. Every
existing site therefore keeps its effective behaviour; seven gain the floor and
nine previously unbounded ones gain a deadline.

Also fixed while here:
- `gpu-amd.sh` had the least-bounded curl usage in the repo — no TLS floor, no
  timeout, no retry. Both calls now go through the wrapper; the `.deb` download
  is retry-wrapped. The listing scrape deliberately is not: `retry()` reports
  attempts on stdout, which is that function's return value.
- `install-k8s.ps1`'s WSL2 here-string had the same two nvidia-container-toolkit
  fetches bare. It cannot source `common.sh`, so it spells the flags out inline
  the way the bootstrap does.

`scripts/install.sh` keeps its seven hardcoded literals: it is the trust root
that fetches `common.sh`, so it cannot source the wrapper.

`CURL_SECURE` stays defined and unchanged for out-of-tree callers, but nothing
in the repo reads it now — the wrapper names the flag itself, so the constant can
never silently reshape every fetch in the installer.

Enforcement: an INTERIM third check in `scripts/check-style.sh` fails on a bare
`curl`. tracebloc/.github#65 already implements this properly (a shell-aware
lexer, not a grep) in a shared reusable workflow, but that workflow is not on
`main` yet and cannot be referenced from here until it is. The check is marked
for retirement the moment this repo adds that caller.

Regenerated `scripts/manifest.sha256` (R8 supply-chain gate).

Found in #399.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ne (Bugbot)

`_fetch_kubectl` had no time bound at all, so routing it through `curl_secure`
handed it the wrapper's default `--max-time 300`. kubectl is a ~50 MB binary, and
this repo already documents (at `_fetch_k3d_release`, same file) that a fixed
ceiling fails a slow-but-healthy link at that size — so the wrapper would have
made every retry fail where the fetch previously completed.

Give both fetches the same `--connect-timeout 15 --speed-limit 1024
--speed-time 60` as the k3d pair. That is also how `curl_secure` knows to skip its
default deadline, and it is strictly better than before: the fetch was previously
unbounded in both directions, so a mid-stream stall hung the step indefinitely.

Audited the other 7 sites that now inherit the 300s default — get.docker.com,
get-helm-3, the Homebrew script, stable.txt, the DMG checksum, the device-plugin
manifest and the amdgpu-install package are all small text/script payloads. The
only large downloads in the repo are kubectl, k3d and the Docker Desktop DMG; the
latter two were already stall-bounded.

Adds a bats test pinning it, since nothing covered `_fetch_kubectl` before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit b1b6dec. Configure here.

`_find_package_name` ran `curl … | grep … | head -1` as a single pipeline and
returned its status. `install-k8s.sh` sources this lib under `set -euo pipefail`,
so that pipeline could kill the installer two different ways:

1. A failed fetch (404, timeout, proxy block) made the command substitution
   non-zero, the caller's assignment inherited it, and `set -e` aborted BEFORE
   the friendly `[[ -z "$name" ]] && error "No amdgpu-install …"` on the next
   line could run. The user got a silent abort mid-GPU-step instead of an
   actionable message, and the RHEL major-version fallback was unreachable for
   the same reason.

2. `head -1` can close the pipe while grep is still writing, so grep takes
   SIGPIPE (141) and `pipefail` propagates that as a pipeline failure even
   though a filename WAS found. It only triggers when the directory index
   exceeds the pipe buffer, so it fails on large mirrors only.

Capture the fetch and the match separately and let neither fail the function,
then take the first match with `${var%%…}` so `head` leaves the pipeline
entirely. The contract is unchanged — filename on stdout, nothing when not
found — so emptiness remains the single signal all three callers already test.

Adds scripts/tests/gpu-amd.bats (first coverage for this lib): the contract,
both hazards, and a caller-shaped regression test under `set -euo pipefail`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka
LukasWodka force-pushed the fix/gpu-amd-pipefail-hazard branch from b1b6dec to fa0f49b Compare July 25, 2026 08:36
@LukasWodka

Copy link
Copy Markdown
Contributor Author

Rebased onto the base branch's round-2 commit (f337c15, the kubectl stall-bound fix) so CI validates the combined manifest rather than a state one commit behind — scripts/manifest.sha256 is a line-per-file list, so both hash updates coexist, and gen-manifest.sh --check reports up to date on the merged result. Re-verified after the rebase: gpu-amd.bats 7/7, setup-linux.bats green, check-style.sh clean, shellcheck --severity=error clean.

bugbot run

@LukasWodka

Copy link
Copy Markdown
Contributor Author

Correction to the "Base branch" note above

I wrote that "GitHub will retarget this to develop automatically" once #400 merges. That is wrong for this repo. GitHub's auto-retarget fires when the base branch is deleted on merge, and tracebloc/client has delete_branch_on_merge: false. So after #400 merges, this PR will keep pointing at a stale, already-merged branch until someone changes it.

Second, this repo squash-merges (every recent develop commit carries a (#N) suffix). After #400 is squashed into one commit, this branch's history no longer matches develop, so a plain retarget would show #400's changes over again.

So the actual sequence after #400 merges is two steps, not zero:

  1. retarget this PR's base to develop, and
  2. rebase fix/gpu-amd-pipefail-hazard onto develop so the diff collapses to just the gpu-amd.sh change and its bats file.

I'll do both as soon as #400 lands — no reviewer action needed beyond merging #400 first. Flagging it here so nobody merges this into a dead branch in the meantime.

Nothing can go wrong by accident, which is the point of the stack: this PR's base is #400's branch, so merging it can only ever land these commits there — never in develop ahead of the TLS fix.

The reason for stacking rather than conflicting still stands: #400 rewrites the exact line this PR rewrites, and the merge-conflict resolution would sit on top of the curl_secure call that #400 exists to introduce.

divyasinghds
divyasinghds previously approved these changes Jul 27, 2026

@divyasinghds divyasinghds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

_find_package_name split into separate captures (index="$(curl_secure …)" || return 0, matches="$(… grep …)" || return 0, first line via ${matches%%$'\n'*}) genuinely fixes both set -e/pipefail hazards:

  • A failed fetch / no-match now returns 0-with-empty output, so callers reach their own [[ -z … ]] && error (and the RHEL major-version fallback is reachable again).
  • Dropping head -1 removes the SIGPIPE-on-large-index failure that only bit on big mirrors.

Contract preserved — the empty return happens before printf, so no bare newline is emitted when nothing is found. First test coverage this lib has had (7 cases), and it's mutation-checked: reverting the function fails exactly the two hazard tests while the contract tests stay green.

⚠️ Merge-order: this is stacked on fix/1252-curl-secure-wrapper (#400), not develop — merge #400 first and GitHub will retarget this automatically. The hazard predates #400; #400 just rewrites the same line to curl_secure.

Reviewed with Claude Code.

@LukasWodka
LukasWodka force-pushed the fix/1252-curl-secure-wrapper branch from f337c15 to 0cefb9c Compare July 27, 2026 07:08
Keep #401's pipefail-safe _find_package_name (drops develop's SIGPIPE-prone `| head -1`); regenerate manifest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 27d9429. Configure here.

@aptracebloc aptracebloc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — the fix itself is approve-worthy, but I'm holding because the PR can't be reviewed or merged in its current shape.

The fix (fa0f49b, _find_package_name in gpu-amd.sh) is correct. It fixes both real hazards of running fetch → grep → head -1 as one pipeline under the entrypoint's set -euo pipefail:

  1. a failed fetch aborting the installer before the [[ -z ]] && error on the next line (and killing the RHEL major-version fallback); and
  2. head -1 SIGPIPE-ing grep on large mirror indexes, failing a lookup that succeeded.

Capturing the fetch and the match separately (each || return 0) and taking the first match with ${matches%%$'\n'*} — so head leaves the pipeline entirely — is the right fix, the contract is preserved (filename or empty, the single signal all three callers test), and gpu-amd.bats adds the lib's first coverage, including a caller-shaped regression under set -euo pipefail that asserts the error path is reached rather than the shell dying. Bugbot clean. 👍

Why I can't approve the branch as-is:

  • Base is fix/1252-curl-secure-wrapper, not develop.
  • CONFLICTING/DIRTY, and the diff is polluted: 29 files / +830, carrying #398 (already on develop), #400 (curl TLS floor), a kubectl-fetch fix, and a "Merge develop into…" commit — for what is really a ~25-line change plus its test. That isn't a diff anyone can meaningfully review or merge.

Recommend: retarget to develop and rebase so the diff is just the ROCm fix (fa0f49b) + gpu-amd.bats (and decide whether the kubectl-fetch fix rides along or splits into its own PR). Once it's a clean single-purpose diff against develop, this is an easy approve — the underlying change is solid.

— drafted with Claude (Opus 4.8), sent by @aptracebloc

Regenerate manifest.sha256 for the merged installer set (idempotent).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit ca65b12. Configure here.

@LukasWodka
LukasWodka changed the base branch from fix/1252-curl-secure-wrapper to develop July 27, 2026 07:53
@LukasWodka
LukasWodka dismissed divyasinghds’s stale review July 27, 2026 07:53

The base branch was changed.

@divyasinghds divyasinghds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approve. #400 merged, this PR auto-retargeted onto develop, and the rebase (fa0f49bcca65b12b) dismissed my prior approval. Re-verified the rebased diff: the _find_package_name fix, the gpu-amd.bats tests, and the manifest are byte-identical to what I approved — the develop rebase changed nothing here.

⚠️ Before merge: get the installer-tests suite to run. It has never run on this PR. installer-tests.yaml only triggers on pull_request to [main, develop, openshift], but this PR's base was fix/1252-curl-secure-wrapper until #400 merged, and the automatic base-retarget doesn't fire a fresh run — so only pii-gate + Bugbot have executed, and the new gpu-amd.bats cases have never been exercised in CI. A single push to the head (e.g. an empty commit or a re-run of installer-tests) will trigger the full suite now that the base is develop. (helm-ci correctly skips — no client/** paths touched.)

Code-wise this is good to go; I just don't want the bats coverage to land unrun.

Reviewed with Claude Code.

Retargeting the PR base from #400's merged branch to develop doesn't fire a
pull_request event, so standard-checks (Unit tests + Lint) never ran on this
head. Empty commit fires synchronize so the required checks run against develop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka
LukasWodka merged commit f63ee1b into develop Jul 27, 2026
31 checks passed
@divyasinghds
divyasinghds deleted the fix/gpu-amd-pipefail-hazard branch July 27, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants