fix(installer): stop the ROCm package lookup from aborting the installer - #401
Conversation
… (backend#1252) `CURL_SECURE` was a bare constant every call site had to splice in by hand, so call sites kept losing it: seven live `curl` invocations ran with no minimum TLS version, including the POST in `verify_credentials()` that carries the client's password. These installs run on customer-managed hosts and behind TLS-inspecting proxies, which negotiate down to whatever the client permits — the reason this repo adopted an explicit floor instead of trusting curl's defaults. Add `curl_secure()` in `scripts/lib/common.sh` and route every fetch in `scripts/lib/*.sh` through it (18 call sites). The wrapper always passes `--tlsv1.2` and supplies default `--connect-timeout 30` / `--max-time 300`. Defaults are injected before `"$@"`, so a call site that wants a tighter bound still wins (curl honours the last occurrence), and a transfer that bounds itself with `--speed-limit`/`--speed-time` gets no injected `--max-time` — a hard deadline would fail a slow-but-healthy link on a large binary download. Every existing site therefore keeps its effective behaviour; seven gain the floor and nine previously unbounded ones gain a deadline. Also fixed while here: - `gpu-amd.sh` had the least-bounded curl usage in the repo — no TLS floor, no timeout, no retry. Both calls now go through the wrapper; the `.deb` download is retry-wrapped. The listing scrape deliberately is not: `retry()` reports attempts on stdout, which is that function's return value. - `install-k8s.ps1`'s WSL2 here-string had the same two nvidia-container-toolkit fetches bare. It cannot source `common.sh`, so it spells the flags out inline the way the bootstrap does. `scripts/install.sh` keeps its seven hardcoded literals: it is the trust root that fetches `common.sh`, so it cannot source the wrapper. `CURL_SECURE` stays defined and unchanged for out-of-tree callers, but nothing in the repo reads it now — the wrapper names the flag itself, so the constant can never silently reshape every fetch in the installer. Enforcement: an INTERIM third check in `scripts/check-style.sh` fails on a bare `curl`. tracebloc/.github#65 already implements this properly (a shell-aware lexer, not a grep) in a shared reusable workflow, but that workflow is not on `main` yet and cannot be referenced from here until it is. The check is marked for retirement the moment this repo adds that caller. Regenerated `scripts/manifest.sha256` (R8 supply-chain gate). Found in #399. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ne (Bugbot) `_fetch_kubectl` had no time bound at all, so routing it through `curl_secure` handed it the wrapper's default `--max-time 300`. kubectl is a ~50 MB binary, and this repo already documents (at `_fetch_k3d_release`, same file) that a fixed ceiling fails a slow-but-healthy link at that size — so the wrapper would have made every retry fail where the fetch previously completed. Give both fetches the same `--connect-timeout 15 --speed-limit 1024 --speed-time 60` as the k3d pair. That is also how `curl_secure` knows to skip its default deadline, and it is strictly better than before: the fetch was previously unbounded in both directions, so a mid-stream stall hung the step indefinitely. Audited the other 7 sites that now inherit the 300s default — get.docker.com, get-helm-3, the Homebrew script, stable.txt, the DMG checksum, the device-plugin manifest and the amdgpu-install package are all small text/script payloads. The only large downloads in the repo are kubectl, k3d and the Docker Desktop DMG; the latter two were already stall-bounded. Adds a bats test pinning it, since nothing covered `_fetch_kubectl` before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit b1b6dec. Configure here.
`_find_package_name` ran `curl … | grep … | head -1` as a single pipeline and
returned its status. `install-k8s.sh` sources this lib under `set -euo pipefail`,
so that pipeline could kill the installer two different ways:
1. A failed fetch (404, timeout, proxy block) made the command substitution
non-zero, the caller's assignment inherited it, and `set -e` aborted BEFORE
the friendly `[[ -z "$name" ]] && error "No amdgpu-install …"` on the next
line could run. The user got a silent abort mid-GPU-step instead of an
actionable message, and the RHEL major-version fallback was unreachable for
the same reason.
2. `head -1` can close the pipe while grep is still writing, so grep takes
SIGPIPE (141) and `pipefail` propagates that as a pipeline failure even
though a filename WAS found. It only triggers when the directory index
exceeds the pipe buffer, so it fails on large mirrors only.
Capture the fetch and the match separately and let neither fail the function,
then take the first match with `${var%%…}` so `head` leaves the pipeline
entirely. The contract is unchanged — filename on stdout, nothing when not
found — so emptiness remains the single signal all three callers already test.
Adds scripts/tests/gpu-amd.bats (first coverage for this lib): the contract,
both hazards, and a caller-shaped regression test under `set -euo pipefail`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b1b6dec to
fa0f49b
Compare
|
Rebased onto the base branch's round-2 commit ( bugbot run |
Correction to the "Base branch" note aboveI wrote that "GitHub will retarget this to Second, this repo squash-merges (every recent So the actual sequence after #400 merges is two steps, not zero:
I'll do both as soon as #400 lands — no reviewer action needed beyond merging #400 first. Flagging it here so nobody merges this into a dead branch in the meantime. Nothing can go wrong by accident, which is the point of the stack: this PR's base is #400's branch, so merging it can only ever land these commits there — never in The reason for stacking rather than conflicting still stands: #400 rewrites the exact line this PR rewrites, and the merge-conflict resolution would sit on top of the |
divyasinghds
left a comment
There was a problem hiding this comment.
Approve.
_find_package_name split into separate captures (index="$(curl_secure …)" || return 0, matches="$(… grep …)" || return 0, first line via ${matches%%$'\n'*}) genuinely fixes both set -e/pipefail hazards:
- A failed fetch / no-match now returns 0-with-empty output, so callers reach their own
[[ -z … ]] && error(and the RHEL major-version fallback is reachable again). - Dropping
head -1removes the SIGPIPE-on-large-index failure that only bit on big mirrors.
Contract preserved — the empty return happens before printf, so no bare newline is emitted when nothing is found. First test coverage this lib has had (7 cases), and it's mutation-checked: reverting the function fails exactly the two hazard tests while the contract tests stay green.
fix/1252-curl-secure-wrapper (#400), not develop — merge #400 first and GitHub will retarget this automatically. The hazard predates #400; #400 just rewrites the same line to curl_secure.
Reviewed with Claude Code.
f337c15 to
0cefb9c
Compare
Keep #401's pipefail-safe _find_package_name (drops develop's SIGPIPE-prone `| head -1`); regenerate manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 27d9429. Configure here.
aptracebloc
left a comment
There was a problem hiding this comment.
Reviewed — the fix itself is approve-worthy, but I'm holding because the PR can't be reviewed or merged in its current shape.
The fix (fa0f49b, _find_package_name in gpu-amd.sh) is correct. It fixes both real hazards of running fetch → grep → head -1 as one pipeline under the entrypoint's set -euo pipefail:
- a failed fetch aborting the installer before the
[[ -z ]] && erroron the next line (and killing the RHEL major-version fallback); and head -1SIGPIPE-ing grep on large mirror indexes, failing a lookup that succeeded.
Capturing the fetch and the match separately (each || return 0) and taking the first match with ${matches%%$'\n'*} — so head leaves the pipeline entirely — is the right fix, the contract is preserved (filename or empty, the single signal all three callers test), and gpu-amd.bats adds the lib's first coverage, including a caller-shaped regression under set -euo pipefail that asserts the error path is reached rather than the shell dying. Bugbot clean. 👍
Why I can't approve the branch as-is:
- Base is
fix/1252-curl-secure-wrapper, notdevelop. CONFLICTING/DIRTY, and the diff is polluted: 29 files / +830, carrying #398 (already on develop), #400 (curl TLS floor), a kubectl-fetch fix, and a "Merge develop into…" commit — for what is really a ~25-line change plus its test. That isn't a diff anyone can meaningfully review or merge.
Recommend: retarget to develop and rebase so the diff is just the ROCm fix (fa0f49b) + gpu-amd.bats (and decide whether the kubectl-fetch fix rides along or splits into its own PR). Once it's a clean single-purpose diff against develop, this is an easy approve — the underlying change is solid.
— drafted with Claude (Opus 4.8), sent by @aptracebloc
Regenerate manifest.sha256 for the merged installer set (idempotent). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit ca65b12. Configure here.
divyasinghds
left a comment
There was a problem hiding this comment.
Re-approve. #400 merged, this PR auto-retargeted onto develop, and the rebase (fa0f49bc → ca65b12b) dismissed my prior approval. Re-verified the rebased diff: the _find_package_name fix, the gpu-amd.bats tests, and the manifest are byte-identical to what I approved — the develop rebase changed nothing here.
installer-tests.yaml only triggers on pull_request to [main, develop, openshift], but this PR's base was fix/1252-curl-secure-wrapper until #400 merged, and the automatic base-retarget doesn't fire a fresh run — so only pii-gate + Bugbot have executed, and the new gpu-amd.bats cases have never been exercised in CI. A single push to the head (e.g. an empty commit or a re-run of installer-tests) will trigger the full suite now that the base is develop. (helm-ci correctly skips — no client/** paths touched.)
Code-wise this is good to go; I just don't want the bats coverage to land unrun.
Reviewed with Claude Code.
Retargeting the PR base from #400's merged branch to develop doesn't fire a pull_request event, so standard-checks (Unit tests + Lint) never ran on this head. Empty commit fires synchronize so the required checks run against develop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Summary
_find_package_nameinscripts/lib/gpu-amd.shran the fetch, the match andhead -1as one pipeline and returned its status.scripts/install-k8s.shsources this lib underset -euo pipefail, so that pipeline could take the installer down two different ways:1. A failed fetch aborted the installer before the friendly error could report it. On a 404, timeout or proxy block the command substitution is non-zero, the caller's assignment inherits that status, and
set -eaborts — before the[[ -z "$deb_name" ]] && error "No amdgpu-install .deb found at …"sitting on the very next line. The operator saw a silent abort mid-GPU-step instead of an actionable message. The RHEL major-version fallback (/rhel/9/after/rhel/9.4/misses) was unreachable for the same reason — the first empty result killed the script rather than falling through.2.
head -1could SIGPIPE grep, failing a lookup that had actually succeeded.head -1closes the pipe as soon as it has its line; if grep is still writing it takes SIGPIPE (141), andpipefailpropagates that as a pipeline failure even though the filename was found. This only triggers once grep's output exceeds the pipe buffer, so it fails on large mirror indexes and passes everywhere else — the worst failure mode to leave in an installer.The fix
Capture the fetch and the match separately, let neither fail the function, and take the first match with
${matches%%$'\n'*}soheadleaves the pipeline entirely:Chosen over sprinkling
|| true, per the ticket discussion:|| trueon the original pipeline would have silenced the fetch failure but left thehead/SIGPIPE interaction in place. The contract is unchanged — the filename on stdout, nothing at all when not found — so emptiness stays the single signal all three call sites already test for, and each one now reaches its own error message.No
set -euo pipefailadded to the lib:scripts/lib/*.shdeliberately have none (the entrypoint sets it atinstall-k8s.sh:44).Tests
Adds
scripts/tests/gpu-amd.bats— the first coverage this lib has had. Seven cases: the three contract cases (finds the file, returns only the first of several, honours the requested extension), both hazards, and a caller-shaped regression that runs the real assignment underset -euo pipefailand asserts the error path is reached rather than the shell dying.Mutation-checked. Reverting the function to its previous form fails exactly the two hazard tests (6 and 7) while the five contract tests keep passing — so the new tests detect these defects, and the contract tests independently confirm the happy path is unchanged.
The large-index test generates 20 000 matches so the SIGPIPE case is deterministic here, rather than depending on a real mirror's index size.
Verification
bats scripts/tests/gpu-amd.bats— 7/7 passbats scripts/tests/*.bats— 427 pass. Three failures (validate_config: valid config passes,un-stamped DEFAULT_REF fails closed,_extract_yaml_value: single-quoted with '' escape) are pre-existing: identical by name on the unmodified base branch, and local-environment artifacts on macOS bash 3.2 — the suite's own notes say Linux CI is the authority.shellcheck --severity=errorand--severity=warningclean;bash -ncleanbash scripts/check-style.sh— style + terminology cleanscripts/gen-manifest.shre-run;scripts/manifest.sha256committed (R8 supply-chain gate)Base branch — please note
This is stacked on
fix/1252-curl-secure-wrapper(#400), not ondevelop, because #400 rewrites the same line to usecurl_secure. Targetingdevelopwould have produced a guaranteed conflict on that line, and a careless resolution could silently drop the TLS floor #400 exists to add. Merge #400 first; GitHub will retarget this todevelopautomatically. The hazard itself predates #400 — it is present ondeveloptoday, just with plaincurl.Found while fixing backend#1252 in #400 and deliberately left out of that PR to keep it behaviour-neutral.
Note
Medium Risk
Touches the GPU install path in the k8s installer; behavior is intentionally narrowed to always exit 0 with empty output on failure, which is well covered by new tests but still affects production ROCm setup.
Overview
_find_package_nameinscripts/lib/gpu-amd.shno longer runs fetch, grep, andhead -1as one pipeline. The index fetch and regex match are captured in separate steps, each allowed to fail without exiting the function (|| return 0), so callers underset -euo pipefailstill get empty stdout instead of a silent abort. The first match is taken with${matches%%$'\n'*}instead ofhead -1, avoiding SIGPIPE/pipefail failures on large mirror listings while keeping the same contract (filename on stdout, empty when not found).Adds
scripts/tests/gpu-amd.batswith seven cases covering happy path, extension filtering, failed fetch, no match, large-index pipefail, and a caller-shapedset -euo pipefailregression. Updatesscripts/manifest.sha256for the changed lib.Reviewed by Cursor Bugbot for commit aeb22fe. Bugbot is set up for automated code reviews on this repo. Configure here.