fix: gate cross-node fetch auth to FlowMesh origin and bound timeouts - #117
Merged
Conversation
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
timzsu
marked this pull request as ready for review
July 27, 2026 15:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Follow-up to #115. That PR fixed one cross-worker image fetch; this PR audits the whole worker/shared surface for the same bug class and fixes the rest.
The class: code paths that diverge between single-node co-located deploys and multi-node cross-node deploys. Co-located, an artifact/result/checkpoint resolves to a local filesystem path — the path exercised in every test and demo. Cross-node (producer and consumer on separate nodes, no shared results storage), the same reference falls back to an HTTP fetch of
{FLOWMESH_BASE_URL}/api/v1/results/.... That fallback is rarely exercised, so it had rotted in three ways:401under enforced auth.FLOWMESH_API_KEY) attached to a non-origin URL a workflow happens to reference, leaking it to an arbitrary host.The invariant this PR enforces everywhere: attach the worker bearer only when the URL is the worker's own FlowMesh origin (
is_flowmesh_origin_url); local paths load locally; external URLs fetch unauthenticated with a bounded timeout.Changes
artifacts.pyresolve_artifactnow sendsauth_headers()only to the FlowMesh origin, so an arbitraryimage_embeddingURL/path can't leak the bearer (fixes b on the VLM embedding hot path).is_flowmesh_origin_urlcompares scheme + hostname + effective port (default http/https 80/443) instead of rawnetloc, so an implicit vs explicit default port is not mistaken for a different origin and auth isn't dropped (fixes a latent a).data.py(image inputs) — the dict{"url": ...}branch now routes only true non-originhttp(s)URLs to the unauthenticated loader; local paths and FlowMesh-origin URLs resolve through the auth-aware artifact loader (fixes a local-path regression that would have HTTP-fetched/local/x.png).checkpoints.py— checkpoint download and HTTP destination gate the bearer to the FlowMesh origin (a/b) with explicit timeouts.data_utils.py— prompt-payload JSONL fetch origin-gates the bearer (a/b).ssh_executor.py— the result-bundlewgetgets a bounded timeout (-T 300 -t 1) (c).Test Plan
bash uv sync --all-packages --group ci uv run pytest tests/worker/test_data_mixin_lineage.py tests/worker/test_artifact_utils.py tests/worker/test_data_utils_prompt_payload.py tests/worker/test_checkpoint_utils.py tests/worker/test_checkpoint_utils_destination.py tests/worker/test_ssh_executor_result_mounting.py -q uv run pre-commit run --files src/worker/executors/mixins/data.py src/worker/executors/utils/artifacts.py src/worker/executors/utils/checkpoints.py src/worker/executors/utils/data_utils.py src/worker/executors/ssh_executor.py Test Result
FLOWMESH_API_KEYset, an external URL receives noAuthorizationheader while a FlowMesh-origin URL receivesBearer; timeouts asserted on every fetch;is_flowmesh_origin_urldefault-port equivalence checked both directions.tests/worker/suite not run locally (GPU/model-download heavy) — deferred to CI.Out of scope (tracked separately)
vllm_executor.pycoercescheckpoint.load.type: httptolocaland, on a node without the checkpoint present, silently initializes from the base-model identifier. That's a missing HTTP-download feature / silent-fallback divergence, not an auth/leak/timeout gap — it needs its own PR (wire up the authenticated checkpoint download, or fail loudly). Not addressed here.Pre-submission Checklist
pre-commit run --all-filesand fixed any issues.uv run pytest tests/passes locally.uv sync --all-packages --group ci --frozen).[BREAKING]and described migration steps above.