feat: Starknet RPC healthcheck probing logic - #4013
Conversation
dd726fe to
bb42402
Compare
7f815e0 to
10891cb
Compare
10891cb to
27da846
Compare
27da846 to
d6b074f
Compare
6e97d0c to
10d8d8a
Compare
d6b074f to
6abd408
Compare
10d8d8a to
006c272
Compare
b59da26 to
c986985
Compare
1805d6f to
93c9ab5
Compare
|
PR title type suggestion: This PR changes only configuration files, build artifacts, and static assets—no source code. The type prefix should probably be Also, minor note: there's a typo in the title ( Suggested title: |
93c9ab5 to
1805d6f
Compare
haiyuechen-nearone
left a comment
There was a problem hiding this comment.
Why are these tests written in an integration test module? They look like unit tests to me
I felt the file was getting too big so moved them to a dedicated file, moving them back into crates/foreign-chain-inspector/src/lib.rs
Adds everything needed to check that a foreign-chain RPC provider is on the network the operator intended, implemented end to end for Starknet: - `expected_chain_identity` per chain in config, plus seeded templates. - A `ChainIdentity` value and `ChainIdentityInspector` trait, implemented for Starknet over `starknet_chainId`. - `FanOut` re-keyed by `ProviderId`, so fan-out failures name the provider instead of a list position, and `FanOut::chain_identities` to ask every provider concurrently under a per-provider timeout. - `probe::probe_all_providers`, reporting one typed status per configured provider, with the chains that have no identity impl yet reported as such rather than silently omitted. Nothing calls the probe yet: node wiring, metrics and the debug endpoint follow. The remaining chains get their own `ChainIdentity` impls after that.
Every failure of a jsonrpsee-backed provider reached the probe as `ClientError`, so a 401, a JSON-RPC error object and an unparseable body all reported `Unreachable` and spent the provider's remaining retries on an outcome no retry can change. `classify_rpc_client_error` classifies the client error by what the provider did; it is opt-in, so `FanOut::extract` keeps the whole error and the signing path is unchanged. Along with it: - `ChainIdentityInspector::canonical_identity` puts the configured identity through the same canonical form as the reported one, so a padded or upper-cased chain id no longer reads as the wrong network. - `AuthTokenUnresolved` names the one setup failure an operator can act on, without carrying an error whose text can embed the API key. - Providers get a short backoff between tries.
`no-use-in-fn` forbids `use` inside a function body. The jsonrpsee error types are aliased because `Error` at module scope is `thiserror::Error`.
The config section promised `MissingExpectedIdentity` for any chain left without an identity, and listed a value for every chain, while only the chains with a probe read the field at all. It now says which chains those are and what the rest report. `canonical_text` stripped only a lowercase `0x`, so an operator writing `0X534E5F4D41494E` got a false `WrongNetwork` from a healthy provider. Both prefixes normalize now. Also: - Throttling signalled as a JSON-RPC error object keeps its retries, rather than counting as a refusal no retry can change. - The identity a provider reports is truncated before it reaches a report that ends up in logs and metric labels. - Classifying a client-setup failure no longer re-reads the environment, so it neither guesses nor copies the token again. - `provider_failure` gets a case per variant, including the ones that report the transaction's own state.
expected_chain_identity became expected_network_fingerprint on the parent branch.
Follows the config field name: ChainIdentity becomes NetworkFingerprint, the trait and its methods follow, and MissingExpectedIdentity becomes MissingExpectedFingerprint.
…cally Also marks the golden transaction route for retirement under #3969.
Six tests with one shape become one rstest, each case keeping the note that justifies it.
…nto the crate Trims the probe status docs, restates the fingerprint trait docs as the implementor contract, and turns every Given/When/Then marker bare, keeping the setup information in helper and constant names. Moves the RPC error classification tests from an integration test file into a unit test module, splits the oversized response message from the malformed one, and points the auth material guard test at a mock so it asserts on a status that carries provider text.
1805d6f to
693b5cf
Compare
netrome
left a comment
There was a problem hiding this comment.
Thanks for updating 🙏 only did a shallow sweep now, but seems like most of my nits have been addressed.
| //! * [`probe::probe_all_providers`] asks each provider for the network it serves and compares that | ||
| //! against the operator's configured expectation. | ||
|
|
||
| pub mod probe; |
There was a problem hiding this comment.
Made me think of https://linux.die.net/man/8/modprobe 😅
anodar
left a comment
There was a problem hiding this comment.
Didn't have a deep look but left few non-blocking comments.
Once we wire this in the node startup, we should ideally also add e2e test for it.
| &self.rows | ||
| } | ||
|
|
||
| /// Only configured chains appear, never reports on a chain the operator did not configure. |
There was a problem hiding this comment.
nit: it's clear from the function name, comment doesn't seem to add much value. Consider dropping it.
| /// | ||
| /// Each provider is tried up to `max_retries` times, `timeout_sec` per try, and only for as long as | ||
| /// the failures stay transient. This returns within the largest configured `timeout_sec * | ||
| /// max_retries`, plus the [`foreign_chain_inspector::RETRY_BACKOFF`] between tries. |
There was a problem hiding this comment.
Not part of your change and non-actionable, just a a thought: I think we don't really need max_retries, timeout_sec to be configurable per chain.
I don't see why would we retry one different times or with different timeouts than the other. I think ideally we should just factor it out into retry parameter constants (of backon::backoff::constant::ConstantBuilder type) and pass it to retry.
| Err(error) => rows.push(ProviderHealth { | ||
| chain, | ||
| provider: provider_id, | ||
| status: setup_failure(&error), |
There was a problem hiding this comment.
We're swallowing actual error here and only surfacing AuthTokenUnresolved/ClientSetupFailed, including the error we get from new_inspector would be lot more helpful.
|
|
||
| /// A provider answers what it likes and the report reaches logs and metric labels, so the length is | ||
| /// capped well clear of the longest real fingerprint: Bitcoin's genesis hash, at 66 characters. | ||
| fn bounded(observed: NetworkFingerprint) -> NetworkFingerprint { |
There was a problem hiding this comment.
Why don't we crop it at NetworkFingerprint object creation and drop this method entirely?
That is, drop derived From and add NetworkFingerprint::new(impl Into<String>)
Part of #4003. Stacked on top of #4017
Probe machinery, Starknet only. Introduces a
ChainIdentityvalue, theChainIdentityInspectortrait, andprobe_all_providers. Nothing calls it at startup yet. Node wiring, metrics and the remaining chains in follow-up PRs.Notes for review
FanOutis re-keyed byProviderId. The names were already available at construction and discarded, so fan-out failures loggedinspector_index = 2rather than naming the provider. The only behavioural change here on the signing path;extract's consensus logic is untouched.chain_identities()is deliberately notextract()'s quorum.extractcollapses dissent intoInspectorResponseMismatch: right for signing, wrong for diagnostics, where each provider's failure must stay visible. Hence a separate method, and a separate trait rather than a method onForeignChainInspector.max_retriesis now honoured on the probing path. The probing logic retries transient failures. Currently still a TODO for the signing path.ForeignChainInspectionError::provider_failure() -> Option<ProviderFailure>is new public API on the inspector crate, so the classification lives next to the variants it matches.Nonecovers the variants that report the transaction's own state: a reverted or non-final transaction is an answer, not a provider fault, and only the probe treats it as anomalous.ProviderStatuscarries no rendered error text.Pathauth splices the API key into the URL and upstream errors interpolate that into theirDisplay; a guard test asserts neither reaches the report.Chains with no identity impl report
ProbeNotImplementedfor the time being. This variant goes away once every chain has probing logic implemented.