Skip to content

fix(orchestrator): re-trigger publishes need distinct message IDs#382

Open
behinddwalls wants to merge 1 commit into
preetam/int/e2e-speculation-parityfrom
preetam/int/retrigger-msg-ids
Open

fix(orchestrator): re-trigger publishes need distinct message IDs#382
behinddwalls wants to merge 1 commit into
preetam/int/e2e-speculation-parityfrom
preetam/int/retrigger-msg-ids

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

The mysql message queue makes publishes idempotent on the (topic, partition_key, id) unique key by documented design (safe publish retries), and rows are only removed by offset-based garbage collection — so a publish reusing a prior message ID is silently coalesced against any not-yet-collected row, including already-consumed ones. The new speculation pipeline reuses IDs in exactly the places that must RE-trigger work, and the Phase-B respeculation e2e run caught all of them at once: prioritize rounds published under the bare queue name meant only the first round per queue ever ran (a batch becoming Speculating a moment later was never promoted, never built, and waited forever), and the dependency-failure fan-out published under the dependent's bare batch ID — the same key the score stage had already used — so the wake-up vanished and the dependent stayed stuck at scored.

Every re-trigger publish now mints an ID that is unique exactly when a new round/wake-up is needed and coalescing when it is not. Prioritize rounds: {queue}/{batch}/v{treeVersion} — every persisted tree change is guaranteed a round (and every prioritizer input change is some tree's version bump), while no-change passes coalesce. Dependent fan-out: {dependent}/dep-terminal/{dep} — one wake-up per (dependent, terminal dependency), self-heal redeliveries coalesce among themselves. Prioritize's build republish: {batch}/v{treeVersion} — promotion rounds are guaranteed delivery, pure heal republishes coalesce.

The terminal self-heal branch now re-fans dependents for every terminal state, not just Cancelled/Failed: Succeeded is how dependents learn a dep landed and dead-path the siblings that bet against it — mergesignal already publishes the terminal batch to speculate on success, so this branch is precisely its landing spot.

make fmt && make test (79/79) — speculate suite updated for the new IDs (per-test versions derived from each test's own Update expectations) and for Succeeded's dependent re-fan. The Phase-B respeculation e2e (next in stack) is the regression test that caught both drops: it goes from permanently stuck at scored to landing.

Stack

  1. feat(speculate): write the speculation tree in shadow mode #353
  2. feat(speculation): route builds through the tree via prioritize #365
  3. feat(speculation): reconcile dead paths and finalize from the tree #371
  4. feat(speculation): cancel batches through the tree via prioritize #379
  5. feat(speculation): optimistic merge finalization behind a confirmed hand-off #380
  6. test(e2e): assert speculation-tree parity on the landing pipeline #381
  7. @ fix(orchestrator): re-trigger publishes need distinct message IDs #382

The mysql message queue makes publishes idempotent on the (topic, partition_key, id) unique key by documented design (safe publish retries), and rows are only removed by offset-based garbage collection — so a publish reusing a prior message ID is silently coalesced against any not-yet-collected row, including already-consumed ones. The new speculation pipeline reuses IDs in exactly the places that must RE-trigger work, and the Phase-B respeculation e2e run caught all of them at once: prioritize rounds published under the bare queue name meant only the first round per queue ever ran (a batch becoming Speculating a moment later was never promoted, never built, and waited forever), and the dependency-failure fan-out published under the dependent's bare batch ID — the same key the score stage had already used — so the wake-up vanished and the dependent stayed stuck at scored.

Every re-trigger publish now mints an ID that is unique exactly when a new round/wake-up is needed and coalescing when it is not. Prioritize rounds: `{queue}/{batch}/v{treeVersion}` — every persisted tree change is guaranteed a round (and every prioritizer input change is some tree's version bump), while no-change passes coalesce. Dependent fan-out: `{dependent}/dep-terminal/{dep}` — one wake-up per (dependent, terminal dependency), self-heal redeliveries coalesce among themselves. Prioritize's build republish: `{batch}/v{treeVersion}` — promotion rounds are guaranteed delivery, pure heal republishes coalesce.

The terminal self-heal branch now re-fans dependents for every terminal state, not just Cancelled/Failed: Succeeded is how dependents learn a dep landed and dead-path the siblings that bet against it — mergesignal already publishes the terminal batch to speculate on success, so this branch is precisely its landing spot.

✅ `make fmt && make test` (79/79) — speculate suite updated for the new IDs (per-test versions derived from each test's own Update expectations) and for Succeeded's dependent re-fan. The Phase-B respeculation e2e (next in stack) is the regression test that caught both drops: it goes from permanently stuck at scored to landing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant