feat(node): gate verify-foreign-tx on available chains via the watch channel

[anodar]

Closes #3910

Followup on #3852

NOTE: this should not be merged until predecessor is not released and deployed (one that registers config to the new endpoint), currently available chains is an empty list.

[anodar]

@claude review

[claude]

Pull request overview

This PR replaces the on-demand ReadSupportedForeignChain RPC-style trait (querying the contract on every verify-foreign-tx attempt) with a watch-channel-driven view of available foreign chains and their participant-ID supporters. A per-Running-state resolver task (spawn_supporters_by_foreign_chain) filters the indexer's TLS-key supporter snapshot against the running participant set and the max ForeignTx reconstruction threshold, and republishes the map as SupportersByForeignChain. The verify-foreign-tx provider now gates both the leader's early path (before consuming a presignature) and the shared execute_foreign_chain_request on this map, with a new mpc_num_verify_foreign_tx_unavailable_chain_rejections counter and an integration test that drops registrations mid-flight.

Changes:

• New crates/node/src/foreign_chain_policy.rs module with spawn_supporters_by_foreign_chain, foreign_tx_reconstruction_threshold, and the SupportersByForeignChain type, plus unit tests.
• IndexerAPI loses the ForeignChainPolicyReader type parameter; foreign_chain_supporters_receiver is now watch::Receiver<Option<ForeignChainSupporters>> (Option distinguishes "no snapshot yet" from "no chain available"). RealForeignChainPolicyReader and the GET_SUPPORTED_FOREIGN_CHAINS view helper are removed.
• VerifyForeignTxProvider and MpcClient drop their ForeignChainPolicyReader generic; chain_is_supported becomes the synchronous ensure_chain_is_available reading from the watch channel, and the leader path checks early to avoid burning a presignature.
• Fake indexer, IndexerAPI, Coordinator, run.rs all updated accordingly. New metric registered.
• New integration test tests/verify_foreign_tx_gating.rs using httpmock to stub Bitcoin RPC and toggle chain availability via the fake contract.

Reviewed changes

Per-file summary

File	Description
Cargo.lock, crates/node/Cargo.toml	Add httpmock as a dev-dependency.
crates/node/src/foreign_chain_policy.rs	New module: resolver task, threshold helper, TLS→participant-id resolution, tests.
crates/node/src/indexer.rs	Drop ReadSupportedForeignChain/RealForeignChainPolicyReader/GET_SUPPORTED_FOREIGN_CHAINS; make foreign_chain_supporters_receiver an Option<...>.
crates/node/src/indexer/foreign_chain.rs	Sender wraps values in Option<...> (None until first successful read).
crates/node/src/indexer/fake.rs	Mirror Option<...> wrapping; expose subscribe_foreign_chain_supporters; drop FakeReadSupportedForeignChain.
crates/node/src/indexer/real.rs	Remove the policy-reader oneshot and initialize the watch with None.
crates/node/src/coordinator.rs	Drop the generic parameter; per-Running spawn of the resolver, feed its receiver into VerifyForeignTxProvider.
crates/node/src/mpc_client.rs	Drop the ForeignChainPolicyReader generic.
crates/node/src/providers/verify_foreign_tx.rs	Store the resolved supporters watch receiver instead of the policy-reader; drop the generic parameter.
crates/node/src/providers/verify_foreign_tx/sign.rs	Replace async chain_is_supported with sync ensure_chain_is_available; check twice (early + main), increment the new metric on rejection.
crates/node/src/metrics.rs	Register MPC_NUM_VERIFY_FOREIGN_TX_UNAVAILABLE_CHAIN_REJECTIONS.
crates/node/src/p2p.rs	Add VERIFY_FOREIGN_TX_GATING_TEST port seed.
crates/node/src/run.rs	Drop the generic parameter on create_root_future.
crates/node/src/tests.rs	Wire the new test module; extend the helper with a caller-supplied bitcoin_tx_id.
crates/node/src/tests/verify_foreign_tx_gating.rs	New integration test asserting gating and the metric increment.
crates/node/src/lib.rs	Register the new module.

Findings

No blocking issues found. A few non-blocking observations:

Non-blocking (nits, follow-ups, suggestions):

• crates/node/src/foreign_chain_policy.rs:1 — SupportersByForeignChain carries HashSet<ParticipantId> values, but the only consumer (ensure_chain_is_available) uses .contains_key(...). The participant-id sets are effectively dead until the follow-up to Publish foreign-chain supporters from the indexer via a watch channel #3852 lands. Since the map is threaded end-to-end just for the key check, consider either (a) noting explicitly in the type-level doc comment that the values are staged for future routing use, or (b) collapsing to BTreeSet<dtos::ForeignChain> and re-adding the value shape in the follow-up. Neither costs correctness today; a one-line comment is enough.
• crates/node/src/coordinator.rs:696 — the comment above spawn_supporters_by_foreign_chain(...) explains an important invariant ("running set retained to resharing survivors (active ∩ prospective)") but does not point at the code that establishes it. A brief pointer to where running_mpc_config.participants is narrowed would make the comment self-serviceable in the future.
• crates/node/src/providers/verify_foreign_tx/sign.rs:126 — worth spelling out in the doc comment on execute_foreign_chain_request that the availability check is intentionally redundant with the leader-side early check (the comment currently lives only at the leader-side call site). Followers hit only this path, so the redundancy is not obvious from a reader landing on execute_foreign_chain_request first.
• crates/node/src/tests/verify_foreign_tx_gating.rs:117 — the 1-second tokio::time::sleep after waiting for the fake-core snapshot is a fixed slack for the in-process fan-out and can flake on slow CI. Since you already have a per-node handle available in IntegrationTestSetup, consider polling one of the node's resolver receivers directly (or the metric) instead of sleeping.
• crates/node/src/foreign_chain_policy.rs:159 — the resolver task uses tracing::spawn; per tracking.rs, this requires being called from a tracked task. The coordinator's Running-state closure is tracked, but a debug_assert! or expect on the tracking context inside spawn_supporters_by_foreign_chain (or a note in the doc) would surface a mistaken call from an untracked location earlier.

✅ Approved

[gilcu3]

Left some comments. Not requesting changes because we probably need some discussion around this, and anyway we are not merging this soon.

My main concern is the use of Option in two places.
The first one if for the supported chains. As far I can see, this is a case that cannot happen in practice, once the node is parked on startup until it gets the first set, which it should anyway have as it already needs the contract state to do anything, and these two things are tied together (both as view methods in the contract). I drafted this alternative in #3966 and it seems to work. Happy to hear if there is a downside.

The other place where I think Option might be avoided is in the thresholds in crates/node/src/foreign_chain_policy.rs. The None branch seems to refer to the case where there are no foreign chain domains, but I think we should handle that explicitly, for example by not running the tasks in that case (or some more explicit method).

[gilcu3]

I think I never considered this case before, in which something is supported and the not supported. It opens the possibility for race conditions. We probably need to have some robust way of handling this.

[anodar]

We had similar discussion in #3853 (comment).

Worst case it either times out (if node dropped support for chain that we thought it supported) or we think it's not supported (if node enables support at the "right" time) and fail early, in which case retry will succeed.

We just also check availability (ensure_chain_is_available) here as a way to reduce that race condition window (although not by much), but not trying to eliminate it. Since all nodes are expected to support all the chains, this is similar to node going down/up race condition.

[netrome]

Speaking of this, it's unfortunate that there's no way to batch multiple view calls into a single atomic request. Feels like it would be a useful feature. I wonder how complex this would be in nearcore to support.

[anodar]

@claude review, check that @gilcu3's comments were addressed.

[claude]

Pull request overview

Follow-up review of the changes since gilcu3's round on 2026-07-24. I verified the four commits that landed after the last review (267cc47, a87bc1f, 37dd1c0) and confirmed the previous feedback was addressed:

gilcu3's feedback → resolution

• Option in the supporters channel / SupportersSnapshotNotReady error → Removed. The real indexer now blocks until the first successful read inside monitor_foreign_chain_supporters and hands the receiver back via a oneshot, mirroring monitor_contract_state. ChainAvailabilityError collapsed back to a ChainNotAvailableError struct.
• Option in threshold / no-ForeignTx-domain case → Handled explicitly at the call site: the coordinator now matches on foreign_tx_reconstruction_threshold(...) and skips spawning the resolver task when there is no ForeignTx domain (crates/node/src/coordinator.rs:704). Nice.
• Metric conflating two failure modes → No longer applicable since the snapshot-not-ready case is gone; description now only covers "chain not available".
• Fixed sleep in the integration test → Removed; the test now polls the supporters channel via a new subscribe_foreign_chain_supporters helper.
• Max threshold across ForeignTx domains → TODO(Revisit foreign chain validation reconstruction threshold in case of multiple foreign chain tx domains #3973) added at both foreign_chain_policy.rs:82 and indexer/fake.rs:231.
• Race between "supported → not supported" (sign.rs:65) — anodar responded with the rationale; no code change needed.

Findings

No blocking issues.

Non-blocking (nits / follow-ups):

• crates/node/src/coordinator.rs:709 — watch::channel(SupportersByForeignChain::new()).1 immediately drops the sender to keep the type uniform. The receiver is only ever .borrow()ed in ensure_chain_is_available, so .changed() is never called and this is safe today; but the pattern is a bit obscure. Consider let (_sender, receiver) = watch::channel(SupportersByForeignChain::new()); receiver with a one-liner explaining that the sender is intentionally dropped (constant empty map).
• crates/node/src/indexer.rs:585 — the doc on foreign_chain_supporters_receiver says it is "seeded with the first successful read before the indexer hands it back", which is only true for the real indexer. The fake indexer initializes it with ForeignChainSupporters::new() (empty). Since only the real indexer's semantics matter for production behavior, this is fine, but the wording could be tightened to "the real indexer seeds it with the first successful read…" so a future test reader isn't misled.
• crates/node/src/foreign_chain_policy.rs:17 — the module doc previously noted "Must be called from a tracked task; dropping the returned AutoAbortTask stops the resolver" (helpful because tracking::spawn panics from an untracked context). The current one-liner drops both. Since spawn_supporters_by_foreign_chain is only invoked from the coordinator's tracked closure it does not matter today, but the constraint is a real footgun if the function is called from elsewhere later. Consider restoring the "tracked task" note.
• crates/node/src/foreign_chain_policy.rs:77 — foreign_tx_reconstruction_threshold is documented as "the max reconstruction threshold across ForeignTx domains, None when no such domain exists" but the TODO(Revisit foreign chain validation reconstruction threshold in case of multiple foreign chain tx domains #3973) notes the max-across-domains choice is provisional. Worth having the doc comment itself point at the follow-up (e.g. "max across ForeignTx domains — see #3973 for the multi-domain design") so a reader landing here understands the "why".
• crates/node/src/tests/verify_foreign_tx_gating.rs:170 — SUPPORTERS_PUBLISH_WAIT = 10s is only a ceiling on the fake core's 1s poll; the loop exits as soon as the empty snapshot arrives, so this is correct. However, the comment "the per-node resolver fan-out from it is in-process and subsumed by the response wait below" is slightly misleading: request_verify_foreign_tx_and_await_response waits for a response, so the "subsumed by" claim is really "if the resolver takes a long time to fan out we'd hit the UNAVAILABLE_RESPONSE_WAIT = 15s timeout and the assertion still passes because both cases produce None." Rewording would help future test debuggers understand what could go wrong here.

✅ Approved

[netrome]

Somewhat shallow review, but overall this looks good to me. Thanks!

[netrome]

I know this predates this PR, but shouldn't we do exponential backoff here? It makes logs cleaner if the node gets stuck on this, and also avoids spamming if we'd get rate limited anywhere.

[anodar]

I did have exponential backoff initially on that, but changed based on reviewers feedback.

#3853 (comment)

[netrome]

Hmm I'm not following the reasoning in that thread.

Here, the contract already has all the required API, so emitting an error seems correct - we don't expect this call to fail.

This reasoning seems to assume the contract lacking the API being the only reason for the call to fail. I see several levels of type-erased errors in the view client, so I wouldn't directly trust this to always hold. Are we sure about this? What about network errors? Or other issues in nearcore? How can we be so confident this won't fail? (I know Kevin raised the point, but I assume you had some understanding of this argument when you changed the code)

[anodar]

How can we be so confident this won't fail?

We can't, and all of the cases you listed will start spamming with error logs.

but I assume you had some understanding of this argument when you changed the code

Only argument against exponential backoff was to save us code complexity. While I don't necessarily agree on it, worst case this will spam 1 log per second so it's not the end of the world. If it keeps spamming long enough (can't read contract state for some reason) we likely have bigger issue (but again not a good reason to avoid backoff).

[gilcu3]

worst case this will spam 1 log per second so it's not the end of the world.

I was not aware of this. As an operator, 1 log per second is not the end of the world, but not too far either as it basically makes the log unreadable after a while

[netrome]

Yeah these kinds of spammy error logs are very annoying, especially considering how clunky the log UX is through dstack, so imo the extra code complexity is worth it. But I wouldn't block the PR on it.

Also, exponential backoff is a well-established pattern in the code base, so I wouldn't consider reusing it in more places adding much complexity. Especially since we still have a loop here. But anyway, do as you please for now. If this happens in production and annoys us we'll revisit and fix it later.

[netrome]

Every time we add one of these direct sleeps without properly injecting time, a kitten dies.

Here it would over-complicate the code to try to shoehorn in a proper time abstraction so that's a separate effort. But anyway.

[anodar]

I had this as config parameter (arguably could have been placed to better place) so that I could set different values in production vs testing code, but changed to hardcoded value as per reviewers suggestion.

Some context in: #3641 (comment), but more was discussed offline on this.

[netrome]

I don't mind the config parameter, I'm just ranting (off topic) about the lack of dependency injection. Time is I/O and I/O should be separated from business logic. When you don't, any code that touches time interactions becomes incredibly hard to test.

For example, if I have:

fn delayed_add(delay: Duration, a: u64, b: u64) -> u64 {
    std::time::sleep(delay);
    a.checked_add(b).expect("overflows aren't the point of this example")
}

This is a very simple function, but how do we write unit tests for this? Yes, we can ignore the delay, but there's no way to control the delay or test it. In more complex systems this becomes a much more real problem since you can't control the timing between components which leads to huge gaps in what you can and cannot test, and buries interesting code behind race conditions and whatnot.

If I instead simply did:

fn delayed_add(sleep: impl Sleep,delay: Duration, a: u64, b: u64) -> u64 {
    sleep.sleep(delay);
    a.checked_add(b).expect("overflows aren't the point of this example")
}

trait Sleep {
    fn sleep(duration: Duration);
}

struct RealTime {}

impl Sleep for RealTime {
    fn sleep(duration: Duration){ std::time::sleep(duration) }
}

I could easily inject a fake sleep in tests that records how long the function actually intended to sleep. Additionally I can introduce wait point to fine grained control over which sleep points are passed and when. This makes time testable, and also make unit tests that don't depend on time instant. You should never have to wait for a timeout in tests.

[anodar]

Fully agree.

In other languages (e.g. Go), I'd stub out foreign_chain_supporters_refresh_interval as a function type variable (maybe even at the package level) and set to different functions depending on production vs testing (pretty close to what you're suggesting). Having time configurable was my way to achieve similar thing in Rust without changing function signature.

I'd rather not refactor this now though as this is existing pattern in the codebase and probably worth something agreeing with others.

[netrome]

Yeah I'm a big fan of functional patterns, although it can sometimes be tricky in Rust if you want to take it too far due to lack of higher order data types etc.

I'd rather not refactor this now though as this is existing pattern in the codebase and probably worth something agreeing with others.

100% agree. My main purpose of the rant is just to create alignment of desired code patterns so we don't leak sub optimal patterns into new modules.

[netrome]

Speaking of this, it's unfortunate that there's no way to batch multiple view calls into a single atomic request. Feels like it would be a useful feature. I wonder how complex this would be in nearcore to support.

[gilcu3]

Thank you for all the fixes!