[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2066
Closed
filimonov wants to merge 2238 commits into
Closed
[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2066filimonov wants to merge 2238 commits into
filimonov wants to merge 2238 commits into
Conversation
…-read caches rejected as alien to CA User rejected the files-cache iteration too: comparing the current backup against the previous one via filesystem metadata is new murky side state, unnatural for content-addressed storage. Final form: consolidation streams every hot file through the ordinary write path (hash under the pool algo -> putIfAbsent/cold-reuse); incrementality and manifest reuse fall out pool-side — the pool is the only index. The trickle warmer needs no bookkeeping (age-threshold crossing is one-shot; cursor loss = harmless re-hash). checksums.txt equality, inode witnesses, and borg-style files caches are all recorded as rejected skip-read shortcuts. Cost stated honestly: one streaming read+hash of the hot tier per consolidation (~minutes/TB daily); the lever if ever measured hot is pluggable hash speed, not bypassing hashing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t to §3) User call: throw out the §7 dedup-skip-HEAD lever as unsafe. Unlike §3 (reader holds a live committed ref pinning the blob), the dedup path producer holds no ref at HEAD time; after §5 an absent .meta reads as Clean even for a fully-deleted blob, so the HEAD is the only thing that distinguishes present from gone. Skipping it would admit a dangle, not elide a bug-net. The dedup HEADs stay; recorded why so a future round does not re-propose it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…ata loss
Optimization-round §5 ("absence-means-Clean blob meta") as specified
re-introduces the data-loss hole fixed on 2026-07-11: its Transition 5
(GcSpareHeal — a spare clears the condemned tombstone) keeps the
condemn-time token t1, so a stale deposed GC leader's pre-CAS
`deleteExact(t1)` deletes the live blob. This is the removed
`clearSparedMeta` "spare = clear" under a new representation; the
2026-07-11 Fix 4 made GC freshness metadata add-only for exactly this.
The §5 gate `CaMetaAbsenceClean.tla` was false-green: single-leader
model whose `GcSpareHeal` atomically empties `queuedDeletes` (line 188),
which the real code cannot do to a deposed leader's captured pre-CAS
delete. A probe with a faithful spare goes RED on
`INV_ABSENCE_NO_QUEUED_DELETE` in 37 states.
Confirmed five ways: the 2026-07-11 report; a gate-fidelity audit
(false-green); a premise-refutation audit (recovery is not only via a
token-displacing resurrect — `Build::adoptEvidence` re-references at
exact t1); the existing add-only guard tests still asserting
Condemned-after-spare; and the mechanical TLC probe. No unsafe code
landed (implementer WIP halted uncommitted, preserved in a stash).
§5 blocked for this round; revive add-only (drop Transition 5) behind a
rebuilt faithful two-leader + adoptEvidence-recovery gate plus the
absent-meta read-path design. A separate, pre-existing `adoptEvidence`
source-dropped-mid-relink exposure is filed for its own investigation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…ality Two per-event wastes on the hot emitter thread of the opt-in content_addressed_log: (a) makeCasEventSink deep-copied every field incl. the full std::map detail because the sink took `const CasEvent &`; (b) the `reason` column was a full String though it is templated rationale. Make CasEventSink take the event by value and emitEvent rvalue-only (a missed call site becomes a compile error, not a silent copy); makeCasEventSink std::moves each field into the log element; `reason` becomes LowCardinality(String). Behavior-preserving. Tests: ReasonColumnIsLowCardinality (schema) and EmitEventMovesSourceIntoSink (source event moved-from after emit). Sweep 985 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
Records the user decision to drop §5 entirely (not revive): spec §5 gets an ABANDONED banner (design-history only), the finding report disposition switches from blocked-pending-revival to abandoned (WIP stash dropped), and the worklog captures the §6 completion and the §5 close-out. The optimization round ships §1–§4 + §6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…e exposure (ClickHouse#42) Read-only reachability investigation. Verdict PLAUSIBLE-NEEDS-REPRO (scoped): all local-source relinks are REFUTED-UNREACHABLE (the source is held as a live DataPartPtr; grabOldParts pins the blob at in-degree >= 1 through the adopt->precommit window). Only fetch-by-relink (remote source) is a residual low-probability exposure (short window; needs a stalled stageManifest PUT across >= 2 GC folds + all-replica part drop + a 2026-07-11 captured deleteExact(t1) firing post-promote). §4 (8fe6331) widened it: it removed the promote-time copyForwardFromCondemned that displaced a condemned adopted leaf's body to a fresh token, the defence that made a captured deleteExact(t1) a TokenMismatch no-op. Distinct from and outside the D4 content-trust envelope (a temporal race that fails silent / fsck-only). Fix (task ClickHouse#43): a liveness re-check at the GC pre-CAS deleteExact site closes the whole deposed-leader same-token class; needs its own TLA gate + second consult. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…point Design approved 2026-07-14: `uuid.txt`, `metadata_version.txt` AND `txn_version.txt` become ordinary manifest tree entries; the per-ref `mutable_files` concept is deleted end-to-end. Committed-part standalone writes (add+overwrite) and surgical removes go through an audited manifest repoint over the existing `publishEntries` sequence. MVCC tmp+rename is short-circuited on atomic-write storages (generic, upstream-candidate), which deletes the B182 eager-dispatch hook — the fork diff outside ContentAddressed/ shrinks below today's. B123 unlink no-op evolves into staged removal marks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…a retention pin Houses tasks ClickHouse#42/ClickHouse#43 in the r/o-replica snapshot-pin design as §8.1: the fetch-by-relink commit-before-release gap (sender fire-and-forget releases the source part before the receiver's adoptPartFromManifest commit → a deep-tail same-token dangle when the receiver's edge-PUT stalls across >=2 GC folds) is closed the right way by reusing the GC retention floor — the fetching replica takes a transient retention pin on the source snapshot for the relink duration. No bespoke handshake (the Poco interserver transport is half-duplex; no reverse ACK). Until the pin lands, ClickHouse#42/ClickHouse#43 is interim-accepted (fsck backstops; deleteExact covers all token-change) — recorded as a comment at adoptPartFromManifest. B66b (relink-into-detached) inherits the same handoff and rides the same pin; B66a (local detached-ref atomicity) is housed alongside as a detached-adjacent but pin-orthogonal item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…Repoint TLA gate) Plan for spec 994507d: repoint primitive over publishEntries (+allow_repoint promote mode matching the existing WRepoint TLA transition), publishStaging carry-forward, MVCC atomic-write short-circuit, writeFile flip, relink self-containment (cookie v2), B123 removal marks, mutable_files schema deletion sweep, freeze special-case removal, docs+validation gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…decommission Extracts the writable-mount tail of `Store::open` into `Store::mountWritable` with a claim policy (`WaitForExpiry` for normal opens, `NoWait` for decommission), adds `readOwnerUuid`, and the victim-impersonation factory per docs/superpowers/specs/2026-07-13-cas-pool-member-decommission-design.md. `NoWait` refuses immediately (no FORCE variant, no wait-and-observe) when the victim's mount lease is not `Claimed`/`FencedSelf`, instead of the bounded reclaim wait a normal open pays. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…`member_decommission` events Task 2 of the pool-member decommission plan: the ORCHESTRATOR that erases every namespace of a DEAD pool member via the ordinary `Store::dropNamespace` writer path (acting as a WRITER, never GC, never inventing a ref transition), over an admin mount obtained through Task 1's `Store::openForDecommission`. Adds `DropNamespaceStats` (`Store::dropNamespace` now reports the committed/precommit counts its removal transaction named), `Core/CasDecommission.h/.cpp` (`DecommissionReport` + `decommissionPoolMember`), and the `MemberDecommission` audit event (`member_decommission`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…vent test Task 2 review found two Important issues in `decommissionPoolMember`: 1. `DropNamespaceStats::precommits` (and `DecommissionReport.precommits_removed`) read 0 for realistic victims. `appendRefOps` hoists `maybeSweepStalePrecommits` at its top for every call, including `dropNamespace`'s own; since the admin decommission mount always mints a strictly-higher `writer_epoch` than the dead victim's, the hoisted sweep reclaims the victim's real precommit bindings in its own transaction before `dropNamespace`'s `build_ops` ever counts them. Fixed by threading a `skip_stale_precommit_sweep` flag through `appendRefOps`, set only by `dropNamespace`'s own call: its removal transaction already names every current precommit (stale or not) via `RemoveNamespace`, making the maintenance sweep redundant on this one path. No other caller is affected. 2. The `member_decommission` begin/namespace_removed/end events were emitted but asserted by no test. Added `CasDecommission.CountsRealisticEpochPrecommit` (seeds a precommit at the victim's own, lower-than-admin `writer_epoch` -- unlike the existing fixture's artificially-high 999999 seed, which dodges this exact race) and `CasDecommission.EmitsMemberDecommissionEvents` (capturing-sink event assertion). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…sk-agnostic B1 proposed a CA-specific manifest_hash field on the ReplicatedMergeTree /parts znode for cross-replica header-divergence detection. Rejected: it would couple ReplicatedMergeTree to one disk implementation and grow the fork's upstream-contact surface. Unnecessary too — fetch-by-relink learns the manifest id in-band (interserver handshake), real divergence is already caught by the stock checkPartChecksumsAndCommit, and manifest-level divergence is a benign, self-cleaning dedup-MISS (01-architecture.md $benign-cross-replica-divergence). Retires a former release GATE. Observability, if ever wanted, is pool-side (fsck/ca-inspect), never a Keeper field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l to read-replica) The fetch-by-relink commit-before-release fix (ClickHouse#42/ClickHouse#43) shares only the GC retention-overlay primitive with the read-only replica feature; owner, payload, lifetime, self-cleaning, and publisher all differ, and it needs none of the reader-mount / snapshot-window / heartbeat / readonly-refresh machinery — so it is independently shippable. New spec captures the converged mechanism: a sender-created, receiver-build-owned, epoch-floor pin published before the part is released (true zero-gap commit-before-release), released by the receiver after its ref commit; bulk write-replica warm-up as a future generalization; interim option-C status; the detached cluster (B66a/B66b). Read-replica §8.1 shrinks to a cross-reference; §5.2 is framed as the shared primitive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Task 3 of the pool-member decommission plan: after namespace erasure (Task 2), reclaim the victim's physical debris. `sweepNamespace` now returns the count of manifest bodies it actually deleted (classified via `DeleteClass::Deleted`), which the manifest-debris phase sums across every eligible old-epoch build prefix under the victim's `cas/manifests/` tree. A new `deleteListedPrefix` helper drains the victim's `staging/<srid>/` and `roots/<srid>/` prefixes by exact-token delete. Fail-close nuance (spec §core "Fail-close"): a per-object failure in the staging/roots drain -- a thrown exception or a `TokenMismatch`/`NotFound` delete outcome -- is recorded as a warning and the sweep continues; this is the only tolerated-and-continue class in the feature. A non-empty `DecommissionReport::warnings` is the signal the (future) slot-deletion phase gates on, so a re-run can finish an interrupted drain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
Add CasDecommission.PerObjectFailureWarnsAndContinuesDrain: a FailingDeleteBackend fixture drives deleteListedPrefix's per-object fail-close path directly -- one thrown exception and one TokenMismatch outcome must each record a warning and let the rest of the drain proceed, rather than aborting the phase or silently counting a miss as deleted. Complements DrainsDebrisStagingAndRoots, which only exercised the clean-drain path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…ent + README registry Approved brainstorm outcome (supersedes codecs proposal v2): - every persisted metadata format becomes human-readable text: JSON control plane (ReadHelpers/JSONEachRow primitives), sorted-NDJSON GC runs (no seek, no blocks; seal-held checksum verified on whole reads), part manifest as JSON descriptor + head -v-style banner payload zone, blob envelope as a 256-byte JSON header line; only payload bytes stay binary - full sweep: refsnaplog (cas_ref_log/cas_ref_snap) and the blob-meta sidecar convert too; inventory corrected against code (no CARS/CART) - codecs move to Core/Formats/ with a physical include-layering rule; Formats/README.md becomes the living registry (bucket map + codec table + evolution rules); protobuf (full runtime + libprotoc) leaves the subsystem - schema evolution: additive = tolerant keys, breaking = v bump + write-down-to-floor, deterministic formats strict + adoption pin Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nue contract Review finding (CRITICAL): the manifest-debris drain (spec §core step 4) did not honor the same fail-close nuance as deleteListedPrefix (staging/ roots). Two defects: (a) sweepNamespace's per-key head/deleteExact was unwrapped, so a transient backend exception propagated straight out of decommissionPoolMember -- staging/roots never ran, no end event, no report; (b) the protection-view-unavailable path (activeManifestKeys throws) only LOG_WARNING'd, never reaching DecommissionReport::warnings -- an unreadable protection view reported success with an undercounted manifest_debris_removed, which would let the future slot-deletion phase (gated on warnings.empty()) delete the mount body over an unconfirmed drain. sweepNamespace gains an optional out-`warnings` vector (nullptr default preserves the exact original behaviour for every pre-existing caller): when non-null, a per-key thrown exception and a protection-view failure both push a warning and the sweep continues, instead of throwing or silently under-reporting. The decommission call site now passes &report.warnings. NotFound/TokenMismatch delete outcomes stay silently spared either way -- that is the normal "a fresh owner reclaimed it" race the periodic sweep expects, not a failure. Added CasDecommission.ManifestDebrisPhaseFailuresWarnAndContinue: a per-key delete-throws case and a protection-view-unavailable case (an isolated namespace with manifest debris but no ref objects, so Task 2's namespace erasure never touches it) both land in report.warnings, and the run completes normally (namespace erasure and the staging drain both still succeed) rather than aborting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
Full TDD plan for the foundation phase: CasFormat move + new FormatIds (RefLog/RefSnapshot/BlobMeta/GcHeartbeat), per-format traits registry, CasTextFormat (JSON micro-vocabulary, JsonObjectReader, header/trailer lines, whole-object zstd arm with cap-before-alloc), shared format test battery + proving instance, Formats/README.md living-registry skeleton. Phases 2-8 get just-in-time plans against this foundation. Spec: docs/superpowers/specs/2026-07-15-cas-codecs-v3-design.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n, fail-close resume
Task 4: the final phase of decommissionPoolMember, strictly last and only
over a clean drain (report.warnings.empty()). A clean drain gracefully
closes the admin Store (Store::~Store's mount-lease keeper stamps the
lease already-expired and folds in the watermark farewell, min_active =
UINT64_MAX -- the terminated state), then deletes the three control
objects (epoch, owner, mount, in that order -- mount last, since it is
the claim/resume anchor). A per-key delete failure here is itself a
drain-incomplete signal: it is appended to report.warnings and the
remaining keys (including the mount) are left alone, so a re-run can
retry. A non-empty warnings set from any earlier phase skips slot
deletion entirely -- the slot stays terminated-but-present, and the
report says so via slot_removed = false plus a LOG_WARNING. Once the
slot is gone, a re-run throws BAD_ARGUMENTS ("unknown pool member"), the
same refusal an srid that never existed gets.
Store gains poolBackendPtr() (an owning BackendPtr, not just a Backend&)
so the slot-retirement phase can keep the backend alive across
admin.reset(). The end MemberDecommission event moves from
EventEmitter{*admin} to calling `sink` directly, since admin is gone by
the time it fires -- this also makes its warnings/slot_removed detail
reflect the truly final state, including a slot-delete failure appended
during retirement itself.
Two pre-existing Task-2 tests assumed a slot survives a clean decommission
run, which is no longer true by design: ErasesAllVictimNamespaces's
tail (re-opening for decommission to verify Removed state) is replaced
with a direct slot_removed assertion, and RerunCountsAlreadyRemoved --
whose "clean run then plain re-run" premise is now structurally
impossible, since a clean run always removes the slot -- is removed;
its assertion is already covered by FailedDrainKeepsSlotThenResumes and
ManifestDebrisFailureKeepsSlotThenResumes (which exercise the same
already-removed-namespace counting via a keep-the-slot-alive resume
instead).
Added CasDecommission.RemovesSlotAndMakesRerunUnknown (the brief's own
test), CasDecommission.FailedDrainKeepsSlotThenResumes (a roots-phase
failure keeps the slot, a disarmed resume finishes and removes it), and
CasDecommission.ManifestDebrisFailureKeepsSlotThenResumes -- a review
follow-up covering the manifest-debris phase specifically, since the
plan's own example only exercises a roots-phase failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…-type, no threshold Amendment (user, 2026-07-16): compression becomes deterministic per type (Never / Always / PinnedRaw). Always = can-grow-large types (cas_ref_log, cas_ref_snap, cas_part_manifest, cas_gc_outcomes) — stored under a .zst key suffix, compressed regardless of instance size, so constructed-key point-GETs stay deterministic and listings are self-describing (zstdcat-able). Always-small types stay bare cat-able raw; deterministic formats (cas_run, cas_fold_seal) stay raw for byte-adoption. The size threshold dies; body magic is validated against the policy (compressed body only legal on Always; raw readable anywhere as the repair path). storedSuffix(FormatId) added to the phase-1 traits registry; compression lands with each format's own cutover instead of a separate step 8 flip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, interpreter with one-row report Task 5 of the pool-member decommission plan: the SQL surface over Cas::decommissionPoolMember (Task 2) / Store::poolBackendPtr, Store::poolConfig (Task 4). - ASTSystemQuery: new Type::CONTENT_ADDRESSED_DROP_POOL_MEMBER, grouped with the two existing CONTENT_ADDRESSED_* enumerators (its magic_enum- derived keyword sequence is CONTENT ADDRESSED DROP POOL MEMBER); formatImpl round-trips `<srid> FROM DISK <disk>` as quoted string literals (an srid is an opaque server-root path, not identifier-shaped, unlike the sibling commands' bare disk target). - ParserSystemQuery: parses the required srid and disk string literals plus FROM/DISK keywords, then ON CLUSTER via the same parseQueryWithOnCluster every other subcommand uses. - AccessType: SYSTEM_CONTENT_ADDRESSED_DROP_POOL_MEMBER (GLOBAL, SYSTEM), grouped with its CONTENT ADDRESSED siblings; 01271_show_privileges reference updated with the corresponding row. - InterpreterSystemQuery: reuses the disk-detection idiom already established by runContentAddressedGarbageCollection/ runContentAddressedGcRebuild (getMetadataStorage() + isContentAddressed() + dynamic_cast, tolerating NOT_IMPLEMENTED from a non-object-storage disk) rather than a fresh dynamic_cast<DiskObjectStorage*> chain, and the MutableColumns/SourceFromSingleChunk one-row-result idiom SYNC_FILESYSTEM_CACHE already uses. Returns one row: srid, namespaces_removed, namespaces_already_removed, committed_refs_removed, precommits_removed, manifest_debris_removed, staging_objects_removed, mountpoint_objects_removed, slot_removed, and a semicolon-joined warnings string. Wired into getRequiredAccessForDDLOnCluster. - Grammar/dispatch-only stateless test (0_stateless/05013_...): a nonexistent disk fails UNKNOWN_DISK before any CA-specific logic runs; real execution needs a CA disk, covered by the integration test. - New src/Parsers/tests/gtest_system_content_addressed_drop_pool_member.cpp: parses-and-round-trips, ON CLUSTER acceptance, and the two required- argument failure shapes -- real grammar coverage runnable via unit_tests_dbms, without needing the full clickhouse server binary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
Task-1 carry-forward, escalated by the Task-4 review: slot-retirement's
crash-safety (deleting epoch/owner/mount in that order, stopping on the
first failure) relies on Store::openForDecommission's owner-anchor-absent
+ mount-lease-present fallback ("partial hand-cleanup: adopt from the
lease", CasStore.cpp) to resolve identity on a re-run after a crash
between the second and third delete. No test exercised this path.
Add CasDecommission.MidRetirementCrashResumesViaMountLeaseFallback:
claims the slot once (as decommissionPoolMember's own first step would),
closes it gracefully, then manually strikes epochKey+ownerKey (leaving
mountKey) -- the byte-identical aftermath of that crash window -- and
asserts a re-run resolves identity via the fallback and finishes
retiring the slot.
Investigation finding worth flagging: claimOwnerOrThrow (CasServerRoot.cpp)
gates the owner-absent path a SECOND, stricter way -- it only re-claims
over a PROVABLY EMPTY data subtree (serverRootSubtreeEmpty:
cas/refs/<srid>/, cas/manifests/<srid>/, roots/<srid>/ all empty). A
victim with a real table trips this: decommission's drain phases never
physically delete ref-log/snapshot debris (that is GC's namespace-cleanup
item's job, on its own later schedule), so the subtree stays non-empty
right after even a fully successful drain. The fallback this test proves
therefore only unconditionally covers a victim with NO namespaces at the
moment of the crash; for a victim that had real tables, the SAME
mid-retirement crash would currently be refused ("identity lost over
existing data") until GC independently clears the residual debris. This
is a real narrowing of "mid-retirement crash-safety" worth a follow-up
decision, not something this task's scope covers fixing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…ber decommission Wires the offline decommission core (`Cas::decommissionPoolMember`, landed in Tasks 2-4) into a `clickhouse-disks` command: `ca-drop-member <srid>` opens the CA disk read-only, resolves its `ContentAddressedMetadataStorage`, and prints the `DecommissionReport` fields (namespaces/refs/precommits/manifest-debris/ staging/mountpoint counts, `slot_removed`, warnings). Pattern follows `CommandCaGcDryRun.cpp`; positional-argument idiom follows `CommandCaInspect.cpp`. Task 7 covers the shared core end-to-end; this is a facade-only, manually smoke-tested addition (no functional pool run here). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…o gtest_Parser.cpp Review intel (verified brief-drift pass): no ParserSystemQuery gtest existed before Task 5, but src/Parsers/tests/gtest_Parser.cpp has the established, reusable parameterized ParserTest fixture (round-trip via formatted-AST equality; every other SYSTEM/DDL parser's coverage already lives there) -- auto-globbed into unit_tests_dbms, no CMake edit needed. Added an INSTANTIATE_TEST_SUITE_P(ParserSystemQuery, ParserTest, ...) block there instead of a bespoke gtest file, matching the sole existing convention for this kind of coverage. This round-trip check caught a REAL bug the previous ad-hoc field-check test missed: CONTENT_ADDRESSED_DROP_POOL_MEMBER was never added to formatImpl's queries_with_on_cluster_at_end set, so `... FROM DISK 'disk1' ON CLUSTER my_cluster` formatted back as `... ON CLUSTER my_cluster 'srv1' FROM DISK 'disk1'` -- ON CLUSTER printed BEFORE the srid/disk arguments instead of after, even though the grammar parses it last (ParserSystemQuery.cpp). The now-superseded gtest_system_content_ addressed_drop_pool_member.cpp only asserted query->cluster == "...", which passes regardless of formatImpl's argument order, so it never caught this. Fixed by adding the type to that set (mirroring CLEAR_FILESYSTEM_CACHE/SYNC_FILESYSTEM_CACHE/CLEAR_QUERY_CACHE, the other commands whose own arguments must print before ON CLUSTER). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
Task 8 of the decommission plan: `04-gc-protocol.md` gains the fence-vs-decommission subsection (fence settles liveness only; the dead member footprint is erased solely by the deliberate `SYSTEM CONTENT ADDRESSED DROP POOL MEMBER` writer-role operation, and `GC` still never invents a ref transition; known fail-closed narrowing documented). `05-formats-and-backend.md` B200 entries flip to implemented-without-roster with the Part IV roster forward-hook. The backlog records B200 done with all landing commits and adds the decommission-under-load scenario-card follow-up. The spec status is stamped implemented. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…P POOL MEMBER`, pool heals
Task 7 of the pool-member decommission plan: a two-node CA/RustFS
integration test modeled on test_content_addressed_shared_pool's
two-node topology (cluster fixture, per-node server_root_id override
files) combined with test_content_addressed_ref_snaplog's RustFS disk
config and read-only clickhouse-disks fsck idiom (MinIO cannot serve CA
pools).
Flow: node2 gets its own table and inserts (data that must NOT
survive), node1 gets its own table (data that MUST survive); hard-kill
node2 (SIGKILL, no graceful farewell); wait for node1 to observe
node2's mount as no longer live via system.content_addressed_mounts;
run SYSTEM CONTENT ADDRESSED DROP POOL MEMBER from node1 and assert the
one-row report (namespaces_removed >= 1, slot_removed = 1, warnings =
''); assert node1's data is untouched and node2's srid is gone from the
mounts table; drive GC to reclaim node2's now-unreferenced content and
poll the RustFS blob prefix back to baseline; assert a read-only fsck
reports dangling=0 and unaccounted=0; re-run the same command and
expect BAD_ARGUMENTS ("unknown pool member").
Execution status: BLOCKED on infrastructure, not a test-code defect.
Ran the new test twice (identical timeout: node1's clickhouse-server
never starts, container running but zero log output). Isolated the
cause with two controls: test_content_addressed_ref_snaplog (single-
node, RustFS) PASSES cleanly; test_content_addressed_shared_pool (the
pre-existing two-node MinIO model test, unmodified) FAILS identically.
The environment cannot currently start ANY two-node integration
cluster, independent of RustFS/MinIO or this test's own config. See
.superpowers/sdd/decom-task-7-report.md for the full diagnostic.
The deferred Task-5 stateless test (05013_system_content_addressed_
drop_pool_member) was run separately via the stateless job selector and
PASSES.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…n integration-test configs The three config files carried ` -- ` inside XML comments, which the XML spec forbids; config merge failed with a SAXParseException before the server opened its log files, so both nodes timed out at cluster setup with an empty logs directory. This was the real cause of the Task-7 integration-run failure — not a broken docker environment: the model test fails for a different, deterministic reason (it mounts the CA disk over MinIO, and the capability probe correctly fail-closes because MinIO does not enforce conditional deletes), which made the two failures look identical at the symptom level. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
node1 mounts the shared pool through two disks (the writable disk plus the read-only fsck window), so `system.content_addressed_mounts` carries one row per disk view for the same srid and the liveness wait got two rows where `assert_eq_with_retry` expected one. Aggregate with `min` so the assert sees a single row regardless of the disk count. First live run after the XML fix reached this assertion (104s in, cluster up, both nodes serving) — the remaining steps run next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…ause; parse TSV `ParserSystemQuery` is not part of `ParserQueryWithOutput`, so `SYSTEM ... FORMAT JSONEachRow` is a syntax error (verified empirically: every SYSTEM query rejects a FORMAT clause). Parse the default TSV row instead, with the column order documented from the interpreter result block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…d-MOVE-kill atomicity never actually verified Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…T queue B199 + S37-oracle before R5 resume Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…shing step 5 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…lost, S39/S36/S37, 20m soak) T4 gate results on the fixed binary (7748419): - dl_probe (original acked-then-lost reproducer, now tracked): LOST(acked-but-absent)=0, submitted=2466 acked=2466 (pre-fix baseline was ~198/1314 lost). - S39 (ClickHouse#37 fence tolerance) at scale=dev: 11/11 verdicts pass, matching the R3 landing run (8e39ea2). scale=ci hit a pre-existing scenario-config bug unrelated to this fix (short_fault_s=15 >= _MOUNT_RENEW_PERIOD_S=10, baked in since the S39 card's creation, ci never previously exercised) -- not a regression, left for a separate fix. - S36 (MOVE PART/PARTITION): 26/26 pass. - S37 (multi-disk policies): 22/23 pass; the one red is the known pre-existing MOVE-PARTITION-under-kill duplicate-row oracle artifact (rows=200 vs expected 100), tracked as GREEN-DEBT (docs/superpowers/cas/BACKLOG.md "S37 chaos-leg oracle bug"), not caused by this fix. - 20-minute soak, seed=42 (the chaos recipe that exposed R4's loss), sync inserts: checkpoint model == observed at every checkpoint (count=813264), fsck clean (unreachable=0 dangling=0), no wedges. This commit also carries the scenario-suite run history that had accumulated uncommitted in this shared worktree from the ongoing prod-scale campaign (Step 9 / ClickHouse#38) since 2026-07-15, appended here alongside the T4 rows -- nothing new was run under this commit besides the dl_probe and soak-phase3 rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…reation, first ci run in DL-fix T4) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…sue draft (not filed), position: GREEN-DEBT then R5 restart from S01 on fixed binary Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…Directory comment updated to the new renameParts-publish reality, RUN_HISTORY chronology Final whole-branch review: READY, zero Critical/Important. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…r overlay, no precommit API) — this working-tree revision was the authoritative text the DL-fix analysis referenced; found uncommitted during session-pause sweep Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…VE-leg PART_IS_TEMPORARILY_LOCKED retry hardening; S39 ci short_fault_s 15->9 + invariant comment S37 oracle fix VALIDATED: dev rerun PASS 23/23 (run2, pre-hardening) — FIRST valid mid-MOVE-kill atomicity verdict: GREEN, no product defect. Pending on resume: one hardened-card S37 validation run + S39 --scale ci rerun (fix applied, unrun). Run1 transient documented: PART_IS_TEMPORARILY_LOCKED race at the MOVE-back leg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…night), R4 10ms query-profiler compose config, scenarios/BACKLOG night findings, stale tmp script deletions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
… GREEN-DEBT S37 23/23 validated, S39-ci/B199 pending; R5 restarts from S01 on fixed binary) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
# Conflicts: # programs/disks/DisksApp.cpp # programs/disks/ICommand.h # src/Disks/DiskObjectStorage/ObjectStorages/Local/LocalObjectStorage.cpp # src/IO/S3/Client.h # src/Interpreters/InterpreterSystemQuery.cpp # src/Parsers/tests/gtest_Parser.cpp # src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp # src/Storages/MergeTree/DataPartStorageOnDiskFull.cpp # tests/queries/0_stateless/02370_lost_part_intersecting_merges.sh
…c references Behavior-preserving cleanup of src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/: - Replace development-diary markers (milestone tokens, `spec §`/RFC pointers, task/phase codenames, dates) with self-contained explanations; the technical content (invariants, orderings, failure behavior) is kept or expanded in place. - Documentation coverage for classes and nontrivial methods; contract comments in headers, mechanics near the implementation. - Mechanical hygiene: duplicate/unused include prunes, a write-only local in `CasProbe`, a dead `if (detail || true)` wrapper in `CasFsck`, single-use `CAS_STRONG_STRING` macro expanded into a plain `RootNamespace` class, Allman reformat of a few one-liners. - Review fixes on top of the base sweep: restored `kRefRecoveryMaxRestarts` (was deleted while still used), corrected an inverted trust-model description in `ContentAddressedExchange.h` (relink promotion trusts adopted refs via the durable manifest edge; no per-blob re-probe), corrected `BlobHashAlgo` persistence claim (byte lives in the source-edge run format, not pool metadata), removed an invented `header_hash` rationale, repaired garbled/mis-indented rewrites, and renamed `<table>` doc-comment placeholders to `<tbl>` to satisfy `-Wdocumentation-html`. Validated: `dbms` and all CAS gtest objects compile cleanly (RelWithDebInfo). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Top-level README: expand the bare layer map into a usable orientation — what the storage is, the five persisted object kinds, write/read/fetch/GC lifecycle walkthroughs, a config example with the observe-only mode, the operator verbs (`fsck`, `ca-inspect`, `ca-gc-dryrun`, `ca-gc-rebuild`, `ca-drop-member`), `system.content_addressed_log`, and how to run the tests (including the comprehensive gtest filter — a bare `Cas*:CA*` silently skips the `RefWriter*` suites). Formats/README: drop internal design-document pointers and the completed migration diary in favor of the timeless statement (the inventory is text end to end), fix the stale part-manifest key shape (`cas/manifests/<ns>/<epoch-hex>-<seq-hex>/<ordinal>.zst`), and split the hex-width rule into fixed-width `UInt128` identities (32 chars) vs algo-length blob digests rendered as `algo:hex`. All command names, key shapes, config fields, job names, and gtest suite names verified against the code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Symbols with zero references anywhere (including tests), verified by grep and by the compiler's -Wunused-private-field after removal: - `CasCodecUtil.h`: `writeU128LE`, `readU128LE`, `writeLenPrefixed`, `readLenPrefixed`, `decodeGuarded` (only the big-endian UInt128 form, `readFixedBytes`, and the name/ref validators remain); pruned the now-unused includes and error-code externs. - `CasTypes.h`: `DigestCodec::digestLen` accessor. - `CasGc`: `readSealedCursors` (private, zero callers) and the `foldManifestEdgesForTest` seam no test uses. - `CasServerRoot`: `SingleWriterSlot::lastRenewTime` and its write-only `last_renew_time` backing field. - `ContentAddressedMetadataStorage`: `serverId` accessor. - `PartFolderAccess`: `PartFolderView::manifestSize`/`publishedAtMs` accessors, the write-only `published_at_ms` field (constructor narrowed accordingly), the `clearForTest` hook no test calls, and the never-produced `LastDecision::Mismatch` enum value (diagnostic-only enum, not persisted). - `CasBlobInDegree.h`: `kSourceEdgeKeySchema` constant (the self-describing key property it documented is stated on `SourceEdgeKeyCodec`). - `CasGcShardPlan.h`: `CoordinatorPlan::shardCount` and the field behind it (the policy object is stateless; its booleans are constant for any `gc_shards >= 1`). Validated: `dbms` and all CAS gtest objects compile cleanly, including the adjusted `gtest_cas_part_folder_view.cpp` constructor call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The predicate family (`hasSingleGlobalFence`, `allowsPerShardFence`, `requiresLeaseForReduce`, `requiresCoordinatorForSeal`, `requiresCoordinatorForFence`) returned constants and was consulted by nothing in production — only its own two gtests read it. Removed the class and those tests. The invariants it encoded are real and stay documented as a free-standing comment in `CasGcShardPlan.h` (single global coordinator fence — a publish into one root shard can protect a blob in any target shard, so a per-reducer fence is unsafe; reducer work is lease-free, the lease being work-dedup only). They are enforced by the actual round code (`Gc::fence`, `CasGcScheduler`), not by this object. The `CasGcShardCoordinator.ShardedFoldRoutesDeltasToOwningShards` test remains: it exercises the live `blobShard` fold routing, not the removed policy object. Validated: `dbms` and the affected gtest objects compile cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The three source-edge row tags (`kEdgeActive`, `kZeroMarker`, `kCondemned`) are bytes of the `cas_run` row format, so they belong in the codec header that owns that format. The separate Primitives header existed only because the tags were originally extracted from the Gc-layer `CasBlobInDegree.h` and needed a home the Formats codec could include; `Formats/` itself satisfies that (`Gc → Formats` is the allowed include direction), and `Gc/CasBlobInDegree.h` already includes `CasRecordStreamFormat.h`, so the merge adds no new include edges anywhere. The sentinel-key contract comment moves with the constants; the README's Primitives inventory is updated. Validated: `dbms` and all CAS gtest objects compile cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Earlier source-layout merges concatenated files, leaving mid-file
`#include` batches (with outright duplicates), second `ErrorCodes` blocks
buried hundreds of lines down, a `namespace fs` alias declared at line
1385, and gratuitous close/reopen churn of the same namespace. Cleaned,
behavior-preserving:
- `Parts/PartFolderAccess.h`: three include batches (with duplicated
entries) consolidated at the top; three `DB::ContentAddressed` blocks
merged to two (the surviving split carries the required `DB::Cas`
forward declaration).
- `Parts/PartFolderAccess.cpp`: mid-file includes and the
`ErrorCodes`/`ProfileEvents`/`CurrentMetrics` extern blocks hoisted to
the top; single `DB::ContentAddressed` block.
- `Pool/CasServerRoot.cpp`: duplicate mid-file includes and a fully
redundant second `ErrorCodes` block deleted; three `DB::Cas` blocks
merged into one. `Pool/CasServerRoot.h`: three `DB::Cas` blocks merged
into one.
- `ContentAddressedTransaction.cpp`: `namespace fs = std::filesystem` and
the `ErrorCodes` block hoisted to the top.
`ContentAddressedTransaction.h`: the old-style nested
`namespace DB { namespace ContentAddressed` wrapper around a single
declaration dissolved; `partFileMustStayBlob` now lives in the file's
existing `DB::ContentAddressed` block.
- `Backend/CasBackend.h`, `Backend/CasRequestControl.cpp`,
`Tools/CasFsck.cpp`, `Parts/PartFolderAccess.cpp`: compact
`namespace DB::ErrorCodes` normalized to the nested form used by the
other 35 files in this directory (and ~15:1 repo-wide).
No namespace of any symbol changed; validated by compiling `dbms` and all
CAS gtest objects cleanly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`Cas` and `ContentAddressed` were synonyms naming two namespaces of one subsystem, so the split signaled nothing: the directory is named `ContentAddressed/` while most of its code was in `Cas`, the adapter namespace covered a small subset (`Parts/`, the transaction buffers, and — inconsistently — `Gc/CasGcScheduler`), and no name collisions existed between the two (verified before merging). The real architectural rule (part-semantics code may use the pool engine, never the reverse — zero reverse references) is carried by the directory layering, not by the namespace. Everything from `DB::ContentAddressed` now lives in `DB::Cas`. The facades stay in `DB` as before (`ContentAddressedMetadataStorage`, `ContentAddressedTransaction`, `IContentAddressedExchange` — they implement or serve `DB` interfaces and carry self-identifying names). DB-pollution audit performed after the merge: the one generic name that sat directly in `DB`, `enum class StagingBackend`, moved to `DB::Cas::StagingBackend`; nothing else CAS-specific remains in the main `DB` namespace except the facades and standard `ErrorCodes` externs. Also merged two anonymous-namespace blocks that were closed and reopened on the next line (`CasInMemoryBackend.cpp`, `CasPoolMeta.cpp`), and replaced an agent-memory slug reference in a `CasPartWriteTxn.h` comment with the invariant it stood for. Validated: `dbms`, `DataPartsExchange.cpp.o`, and all CAS gtest objects compile cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`ShardIncarnation` had zero users — not even the tests whose suite name echoes it (`gtest_cas_gc_shard_incarnation.cpp` tests live discovery and newborn-precommit behavior and never instantiates the type). It is a leftover of the retired per-shard incarnation-stamp design; incarnation ordering is carried by the strictly-increasing `RefTxnId` today. Also removed a tombstone comment in `gtest_cas_pool.cpp` describing already-deleted shard-birth-incarnation tests. Validated: `dbms` and all CAS gtest objects compile cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The old layout hid the concepts: `CasBlobHasher.{h,cpp}` contained no
`BlobHasher` class (it mixed the algorithm vocabulary with the streaming
buffers), the digest value family was buried in `CasTypes.h`, and
`CasXXH3.h`'s single class was named `Xxh3Streamer`.
New layout — file = concept:
- `Primitives/CasBlobDigest.{h,cpp}` — blob identity in one place:
`BlobHashAlgo` (+ name/width/config parsing), `BlobDigest`,
`DigestCodec`, `BlobRef`, `codecFor`, `blobHexOf`/`blobIdOf` (moved
from `CasTypes.h` and the old `CasBlobHasher.h`). Dependency-light:
`CasTypes.h` now includes this instead of pulling `IO/WriteBuffer.h`
into every CAS translation unit.
- `Primitives/CasBlobHashingWriteBuffer.{h,cpp}` (ex `CasBlobHasher`) —
only the streaming machinery: `IBlobHashingWriteBuffer` (renamed from
the confusable `IHashingWriteBuffer`), `makeBlobHashingWriteBuffer`,
`blobHashHexOneShot`.
- `Primitives/CasXxh3Streamer.h` (ex `CasXXH3.h`) — named after the
class it defines; still the isolated vendored-xxHash system-header
wrapper.
Includers re-pointed by what they actually use (six vocabulary-only,
three buffer users); no behavior change anywhere.
Validated: `dbms` and all CAS gtest objects compile cleanly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sumer The per-round tally of failed per-hash freshness-meta ops was folded into `RoundReport` but never mapped into `GcRoundLogRecord`, so it reached no system table and had no reader (tests included). The same failures already increment `ProfileEvents::CasGcMetaWriteAnomaly` (which reaches the GC log row via the per-round profile-events snapshot) and are logged via `tryLogCurrentException`, so no observability is lost. Removed the report field, the backing `meta_anomaly_count` atomic, its per-round reset and fold, and the now-unneeded `this` lambda capture. The operator-signal semantics the field's comment carried (a persistently non-zero rate means the writer's meta point-read gate is drifting from the ledger) moved into the `CasGcMetaWriteAnomaly` profile-event description. Validated: `dbms` and all CAS gtest objects compile cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt hb owners
The steal gate's liveness check compared the observed `gc/hb` sequence
against the previous observation while matching the hb OWNER against the
CURRENT LEASE owner:
hb_alive = has_observation && hb.owner == lease.owner && hb.hb_seq > last_seen_hb_seq
`hb_seq` values are only comparable under the same hb owner — which is
exactly why `last_seen_hb_owner` was stored — but the stored owner was
never read (the half-finished check this fixes). The hazard is reachable:
a deposed leader's heartbeat thread keeps pulsing (`i_am_leader` resets
only on its next round) with `owner = itself`, and `pulseHeartbeat`'s
losing CAS writes vanish silently, so a zombie old leader can keep
`gc/hb.owner` pointing at itself while the live new leader is pulsing
too. On both of a follower's ticks `hb.owner != lease.owner` then made
`hb_alive` false, and with the new leader's lease tuple frozen mid-round
the follower stole the lease from a live, pulsing leader.
The gate is now observation-symmetric with the frozen-lease check: ANY
movement of the observed `(owner, hb_seq)` pair between the contender's
two ticks is proof of life, and `hb_seq` is compared only under the same
remembered hb owner (an owner change re-arms the window). Liveness is
preserved — only leaders pulse and a deposed leader stops within one round
interval, so a genuinely dead leader still gets stolen from one window
later at most.
Failing-first regression test
`CasGcLease.StaleOwnerHeartbeatDoesNotEnableFalseSteal` reproduces the
zombie-masking interleaving deterministically (steal confirmed on the old
predicate) and pins both the no-false-steal and the still-steals-when-
frozen halves.
Validated: full corrected CAS gtest battery 906/907 — the one failure is
the pre-existing tracked `RefWriterRecoverySeal` F3-1a fragility
(`docs/superpowers/cas/BACKLOG.md`), unrelated to the lease protocol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
filimonov
changed the title
filimonov
marked this pull request as draft
July 17, 2026 17:24
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
content addressable storage - draft PR
Documentation entry for user-facing changes
TBD.