Skip to content

chore: improve unsub failure and retry logic - #9678

Open
sahar-fehri wants to merge 9 commits into
mainfrom
chore/enhance-unsub-failure-cleanup-logic-ohlcv
Open

chore: improve unsub failure and retry logic#9678
sahar-fehri wants to merge 9 commits into
mainfrom
chore/enhance-unsub-failure-cleanup-logic-ohlcv

Conversation

@sahar-fehri

@sahar-fehri sahar-fehri commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fix OHLCVService unsubscribe cleanup so failed WebSocket unsubs no longer leak gateway subscription slots and block future OHLCV streams.
  • Flush grace-period channels immediately when subscribing to a different asset/interval, while keeping same-channel grace reuse for rapid back navigation.
  • Retry failed unsubs with backoff (1s → 2s → 4s) and call forceReconnection only after retries are exhausted.

Explanation

Current state: When #performUnsubscribe failed, OHLCVService removed the channel from local tracking before the gateway unsub succeeded. That left orphaned server-side subscriptions, which could exhaust the gateway's 2 market-data subscription cap and cause later subscribes to fail (clients fall back to REST /latest polling).

Solution:

  1. Delete-on-success — keep channel entries until WS unsub succeeds.
  2. Grace-period flush — when subscribing to a different channel, immediately unsub other channels in grace/failed-cleanup instead of holding two live subs for up to 3s. Same-channel grace reuse also flushes other channels first (Bugbot fix).
  3. Retry + last-resort reconnect — on unsub failure, retry with backoff; after 3 retries, call BackendWebSocketService:forceReconnection to reset server subscription state.

Same-channel grace (3s) is unchanged so users returning to the same token/interval within 3s can reuse the existing subscription.

Important: forceReconnection affects the shared WebSocket

BackendWebSocketService is a single shared connection on mobile, used by multiple services (not OHLCV-only):

Consumer Purpose
AccountActivityService Real-time account balance / activity push
OHLCVService Chart OHLCV streaming
AssetsController (indirect) Forwards AccountActivityService:balanceUpdated

Calling forceReconnection() closes and reconnects that entire WebSocket. On disconnect it clears all local subscription state; on reconnect each service resubscribes what it needs:

  • AccountActivityService — marks chains down on DISCONNECTED, then resubscribes the selected account on CONNECTED (this service already uses forceReconnection for its own cleanup today).
  • OHLCVService — only resubscribes channels with refCount > 0 (active chart consumers). Failed-cleanup / grace entries (refCount === 0) are dropped on reconnect and are not resubscribed.

When does OHLCV trigger it?

Only as a last resort, after unsub has failed 4 times total (initial grace-period attempt + 3 retries at 1s / 2s / 4s). Typical navigation (leave chart, switch token, return within grace) uses delete-on-success, flush, or grace reuse — not force reconnect.

Rough timeline after user leaves a chart with a stuck unsub: ~10s of background retries before reconnect fires.

User-visible impact

Scenario Impact
User left chart, unsub stuck → force reconnect (Test 5) Brief (~1–2s) WS gap for account activity push; chart unaffected (hook unmounted). OHLCV does not resubscribe.
User has chart open when reconnect fires (unlikely — retries run after refCount === 0) Brief WS gap; OHLCV resubscribes active channel; mobile hook may poll REST briefly during staleness window.
Normal chart navigation No force reconnect

Why we still use it

  • A permanently leaked market-data sub hits the gateway 2-sub cap and can break all OHLCV streaming on that connection — worse than a one-time reconnect blip.
  • forceReconnection is an existing, shared primitive (introduced in fix(core-backend): reconnection logic #6861; AccountActivityService already relies on it).
  • Retries are attempted first; reconnect is the escalation path only when repeated unsub attempts fail.

Open question for reviewers: If the cross-service reconnect blast radius is unacceptable, alternatives are (a) gateway-side sub cleanup API, or (b) OHLCV-only reconnect/isolation — neither exists today on the shared BackendWebSocketService.

Fixes

References

  • Related mobile investigation: UNSUBSCRIBE_LEAK_FIX_PLAN.md in metamask-mobile
  • Gateway behavior confirmed with backend: steady state is 1 live OHLCV sub per connection; cap of 2 is a race buffer for unsub→sub overlap
  • BackendWebSocketService.forceReconnection()fix(core-backend): reconnection logic #6861

Test plan

  • yarn workspace @metamask/core-backend run jest --no-coverage packages/core-backend/src/ws/ohlcv/OHLCVService.test.ts (43 tests)
  • Manual (mobile, debug build): unsub retry — fail 4×, succeed on 5th
  • Manual: same-token return within grace — reuse WS sub, no new subscribe
  • Manual: token switch within grace — flush old channel immediately
  • Manual: grace reuse flushes other channels first (Bugbot scenario)
  • Manual: exhausted retries → forceReconnection → WS disconnected / connected cycle
  • Manual: rapid token hopping on TDP — verify third subscribe succeeds (no limit would be exceeded)

Checklist

  • I've updated the test suite for new or updated code as appropriate
  • I've documented the shared-WebSocket impact of forceReconnection in code (JSDoc) — PR description above; happy to add JSDoc in follow-up if reviewers want it in-tree
  • I've communicated my changes to consumers by updating changelogs for packages I've changed
  • I've introduced breaking changes in this PR and have prepared draft pull requests for clients and consumer packages to resolve them

Note

Medium Risk
Touches live WebSocket subscription lifecycle and can trigger shared forceReconnection, briefly affecting other consumers on the same connection; mitigated by delete-on-success, flush, and retries before reconnect.

Overview
Fixes OHLCV chart streaming getting stuck when gateway market-data unsubscribe fails by keeping channel entries until the server unsub succeeds, instead of deleting them early and leaking subscription slots.

Navigation behavior changes: subscribing to a different asset/interval immediately flushes other channels still in grace or failed cleanup (same-channel grace reuse for back navigation is unchanged). Re-subscribe paths also flush other pending channels first.

Resilience: failed unsubs go through cockatiel backoff (1s → 2s → 4s, four attempts total), emit subscriptionError, and call BackendWebSocketService:forceReconnection only when retries are exhausted. Retries and grace timers are aborted on destroy or when the channel is subscribed again; reconnect drops refCount === 0 entries instead of resubscribing them.

Adds cockatiel to @metamask/core-backend and expands OHLCVService tests for flush, retry, reconnect, and destroy paths. Changelog also records an unreleased breaking alignment of V6_DEFI_POSITION_TYPES (no code changes for that in this diff).

Reviewed by Cursor Bugbot for commit 54e16dd. Bugbot is set up for automated code reviews on this repo. Configure here.

@sahar-fehri
sahar-fehri marked this pull request as ready for review July 28, 2026 13:39
@sahar-fehri
sahar-fehri requested review from a team as code owners July 28, 2026 13:39
Comment thread packages/core-backend/src/ws/ohlcv/OHLCVService.ts
}
}

#scheduleUnsubscribeRetry(channel: string): void {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can handle the retries to TanStack?

refCount: number;
gracePeriodTimer?: ReturnType<typeof setTimeout>;
retryTimer?: ReturnType<typeof setTimeout>;
retryCount?: number;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially TanStack would remove this kind of trackers and all underlying logic. (gracePeriodTimer, retryTimer, retryCount). I'm not sure about refCount

@Prithpal-Sooriya Prithpal-Sooriya Jul 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed here, maybe we can move away from imperative retry by using either TanStack or cockatiel.

import { ExponentialBackoff, handleAll, retry } from 'cockatiel';
import { QueryClient } from '@tanstack/query-core';

const testFn1 = async () => {
  const result = await retry(handleAll, {
    maxAttempts: 3,
    backoff: new ExponentialBackoff({ initialDelay: 200, maxDelay: 5_000 }),
  }).execute(async () => await doSomethingThatCanFail());
}

const testFn2 = async () => {
  const result = await new QueryClient().fetchQuery({
    queryKey: ['ohlcv-disconnect', someIDIfWannaMakeUnique],
    retry: 3,
    queryFn: () => doSomethingThatCanFail(),
  });
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you guys check this refactor here 74a7e01
i went with cockatiel

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 43628d8. Configure here.

Comment thread packages/core-backend/src/ws/ohlcv/OHLCVService.ts

@Kriys94 Kriys94 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slight preference for re-using TanStack, but I'm also good with the implementation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants