Skip to content

docs(design): operator-prepaid attestation storage - #4011

Merged
barakeinav1 merged 20 commits into
mainfrom
3972-operator-prepaid-attestation-storage
Aug 3, 2026
Merged

docs(design): operator-prepaid attestation storage#4011
barakeinav1 merged 20 commits into
mainfrom
3972-operator-prepaid-attestation-storage

Conversation

@barakeinav1

@barakeinav1 barakeinav1 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes #3972.

Design doc only — no code. Implementation is tracked in #4015.

  • One prepayment buys one grant — permission for a node account to store one attestation entry. A grant returns when the entry it paid for is reclaimed, so it is a slot the operator keeps rather than a per-attestation charge. An operator prepays for the nodes they run plus a spare, typically two: the live node and the one they are migrating to. No NEAR is ever refunded.
  • A counter, not a balance. The contract stores how many unused grants an account has, never how much anyone paid. That removes the arithmetic, and because grants are denominated in entries a fee change cannot strand an operator whose deposit is suddenly worth less than an entry.
  • Payment and submission must be separate transactions. A function-call key cannot attach a deposit — not even inside a meta-transaction, since validate_delegate_action_key applies the same check to inner actions — and report_data binds the quote to env::signer_account_pk(), so an operator cannot submit on the node's behalf.
  • Check early, consume late. The grant is checked before any verification, so an ungranted call never reaches the Mock checks or a verify_quote round trip, and consumed at insert, so a failed attestation consumes nothing. That also keeps refactor(contract): drop fail_attestation_submission now that the deposit is gone #3991 unblocked: no charge has to survive a failing callback.
  • Multi-node and migration need no special rule — prepay again. There is no cap constant, no participant-status check, and no assumption about how many nodes share an account.
  • Records the rejected alternatives with their evidence: full-access node keys, meta-transactions, operator-submits-on-behalf, gating Attestation::Mock, per-account entry caps, the governance gate, refundable bonds, and an owner-callable release method.

Fee is 0.02 NEAR, ~2.7× the ~0.0073 floor (604-byte worst-case entry plus its ~130-byte grants-map row).

Implementation notes captured for whoever picks this up: it needs a state migration for both the grants map and the Config fee field, and rule 3's per-entry write means clean_invalid_attestations's gas budget has to be re-validated.

Separately noted in the doc: docs/running-an-mpc-node-in-tdx-external-guide.md still tells operators the call "will incur a cost (TBD, XXX NEAR)" and points at the now-closed #903. Stale on main today, independent of this design.

@barakeinav1
barakeinav1 force-pushed the 3972-operator-prepaid-attestation-storage branch 2 times, most recently from 575f9ab to 4a16377 Compare July 30, 2026 09:02
Design for #3972: move the storage cost of a stored attestation entry off the
contract's balance and onto whoever onboards the node, by adding a single
operational step in which the operator funds that storage.

Payment and submission have to be separate transactions. A function-call access
key cannot attach a deposit -- not even inside a meta-transaction, since
validate_delegate_action_key applies the same check to inner actions -- and
report_data binds the quote to env::signer_account_pk(), so an operator cannot
submit on the node's behalf. The operator therefore credits a per-account balance
with a deposit-capable key, and the node keeps self-submitting with its own
restricted key.

The charging rules are evaluated read-only at the top of submit_participant_info,
so an unfunded submission is rejected before any verification or cross-contract
round trip, and debited authoritatively at insert, where a failed attestation
stores nothing and so charges nothing. That also keeps #3991 unblocked: no charge
or refund ever has to survive a failing callback.

Migration and key rotation stay free and bounded by state that already exists --
ongoing_migrations holds one declared destination per participant, so a
participant can hold at most two entries and a non-participant none -- rather
than by a new cap constant.

Also records the alternatives that were rejected with their evidence, and states
the one-node-per-account assumption explicitly, since the contract does not
enforce it.
@barakeinav1
barakeinav1 force-pushed the 3972-operator-prepaid-attestation-storage branch from 4a16377 to 6d38389 Compare July 30, 2026 09:03
Reynaldo and Marten pushed back on the storage-credit accounting as more general
than the problem needs, and the follow-up meeting settled on a counter: one
prepayment buys one grant to store one attestation entry, operators prepay again
per extra node, no refunds.

The counter is not just a substitute for the balance -- it removes three things
from the previous draft. Migration no longer needs a free-entry rule gated on
ongoing_migrations, there is no participant-status check, and the
one-node-per-account assumption is gone entirely, since an operator wanting two
nodes simply prepays twice. That also fixes the case Reynaldo raised of an
operator testing several nodes at once, which the migration rule handled badly.

Grants are denominated in entries rather than NEAR, so re-pricing after a
storage-price change cannot strand an existing operator -- a balance model has to
handle exactly that.

Moves the storage-credit design to Alternatives with the reasons it was rejected,
marks every operator step as existing except the prepayment (the off-chain
attestation check read as a new requirement, which it is not), and records the
open question of whether a reclaimed entry should return its grant.
Settle the three decisions left open after review:

- A grant is capacity, not a consumable ticket: reclaiming an entry returns the
  grant to the account that owned it. Without this an operator testing several
  nodes in sequence pays per node even though the contract reclaimed each slot
  (raised by gilcu3). One increment at the contract's single removal site, and no
  NEAR moves, so the no-refunds decision is untouched.
- Delete a counter's row when it reaches zero so the map does not accumulate rows
  for accounts holding nothing.
- Do not confiscate grants when a node leaves the participant set. Kicks are
  often temporary, and taking prepaid capacity would force an operator to pay
  again merely to rejoin.

Also state what the fee actually has to cover, which was missing: the worst-case
attestation entry (604 bytes) plus the grants-map row it creates (~130 bytes),
about 0.0073 NEAR together, plus headroom for future layout growth so a schema
change cannot leave issued grants under-funded. 0.02 NEAR gives ~2.7x. And state
plainly that no NEAR is ever returned -- recycling gives back capacity, never
money, and there is no withdrawal path.

The security argument is extended to show recycling does not weaken the bound:
a grant returns only once the entry it paid for is gone, so entries held never
exceed grants bought.
'An operator running two nodes prepays twice' did not say why anyone would. Give
the two real cases: a backup node, and the second node during a migration.
…ragraph

The background implied contract-funded storage started with #3940. It did not:
the deployed 3.13.0 contains charging code that never collects, because it reads
the storage delta before the write is flushed, and #3714 -- the only version that
genuinely required a deposit -- never shipped. So this has been true of every
deployed version, and #3940 only made it intentional. That matters because it
means the drain is not a regression to be reverted but a gap that was always
there.

Also split the run-on sentence about the security implication into three bullets:
unlimited entries, roughly 7x cost asymmetry, and the contract being unable to
write state at all once its balance is gone.
…otal

attestation_grants(account_id) -> u32 did not say which number it returned. It is
available grants -- bought minus those currently backing an entry -- since the
counter is incremented on prepay, decremented on insert and incremented again on
reclaim. Rename it available_attestation_grants and spell out the relation.

Lifetime total is deliberately not stored: the contract enforces the invariant by
keeping the counter non-negative, so neither 'bought' nor 'entries held' has to be
tracked separately and nothing in the design reads them.
Three paragraphs to say: check the grant before verifying, consume it on success,
consume nothing on failure. Cut to two, keeping only the two things that are not
self-evident -- why both steps exist (a Dstack callback lands later, so the early
check cannot be the enforcement) and the #3991 consequence.
…ionale

Recycling was stated in passing inside the state description and the charging
rules, which under-sold the single property that most changes how the scheme
feels to use. Put it in the opening paragraph, add it as a goal, and give it two
named reasons instead of one:

- fairness, the original argument -- the contract got the storage back, so
  charging again charges twice for the same thing;
- operator experience -- prepay for the capacity you actually run, three grants
  for three nodes, and then stop thinking about it. Nodes can be torn down,
  re-provisioned, rotated and migrated indefinitely on the same grants. Without
  recycling every re-provisioned node is a fresh purchase and a fresh thing to
  remember.
@barakeinav1
barakeinav1 marked this pull request as ready for review July 30, 2026 12:40
Copilot AI review requested due to automatic review settings July 30, 2026 12:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a design-draft document describing an “operator prepays attestation storage” model intended to address #3972 by shifting attestation-entry storage costs off the contract balance while preserving deposit-less self-attestation from function-call keys.

Changes:

  • Introduces a grant-based prepayment model (prepay_attestation_storage) and outlines the intended charging, reclaim, and UX flows.
  • Documents API surface, state additions, sizing rationale, rollout steps, and testing plan.
  • Records alternatives considered (meta-tx, full-access keys, credit accounting, caps, governance gating) and why they were rejected.
Comments suppressed due to low confidence (2)

docs/design/operator-prepaid-attestation-storage.md:76

  • This table entry says any remainder of the attached deposit is kept. That conflicts with #3972’s acceptance criteria calling for any excess over the required flat deposit to be refunded (even if the refund happens in the prepay step rather than on submit_participant_info). Either update the design/issue to match, or change the spec here to refund the remainder.
| `prepay_attestation_storage(account_id: AccountId)` | `#[payable]` | Grants `floor(attached_deposit / fee)` entries to `account_id`. Rejects below one fee. Any remainder is kept. Permissionless — anyone may prepay for any account. |

docs/design/operator-prepaid-attestation-storage.md:120

  • The PR description says the design assumes “one node per NEAR account” (not enforced by the contract), but the design text here says it makes “no assumption about how many nodes share an account.” Please reconcile this by either explicitly stating the operational assumption (while noting the contract doesn’t enforce it) or updating the PR description.
Migration and multi-node operators need no special rule: an operator who wants a second live node prepays a second grant. That is the whole mechanism, and it is why this design needs no cap constant, no participant-status check, and no assumption about how many nodes share an account.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- **Fairness** — the fee bought storage. Once the contract has that storage back, charging again for the next entry would be charging twice for the same thing.
- **Operator experience** — an operator prepays for the capacity they actually run (three grants for three nodes, say) and is then done with it. Nodes can be torn down, re-provisioned, rotated and migrated indefinitely without another prepayment, as long as no more than three entries are held at once. Without recycling, every re-provisioned node is a fresh purchase and a fresh thing to remember.

**No NEAR is ever returned.** Recycling gives back capacity, never money: a deposit is consumed permanently the moment it is made, and the only thing that ever comes back is the right to store another entry. There is no withdrawal path and none is planned.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider tightening the wording to distinguish "no withdrawals / redeem" from "refund any overpayment in the prepay call", so the document is internally consistent and aligned with #3972.

Done — the two were conflated. The doc now states that "No NEAR is ever returned" covers both cases: no withdrawal method, and no refund of an overpayment. It also says why keeping the remainder deviates from the contract's usual require_deposit + refund_to pattern — a grant is a discrete unit, so the leftover is at most one fee short of the next grant, and a transfer path for sub-0.02 NEAR dust is not worth it.

On the #3972 alignment: the issue was the stale side, not the doc. Its first acceptance criterion still required refunding excess, which the design meeting overrode — that AC is now updated, along with three others that had drifted from the agreed model.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my reply above: the design changed rather than the wording.

prepay_attestation_storage now takes an explicit grant count and requires an attached deposit of exactly fee × grants, rejecting anything else:

prepay_attestation_storage(account_id, grants)   #[payable]

So there is no remainder to keep, and nothing to reconcile against the require_deposit + refund_to convention — the case your finding was about no longer exists. My earlier reply justified keeping the remainder; disregard that reasoning. "No NEAR is ever returned" now means only that there is no withdrawal method.

@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

Pull request overview

Doc-only PR adding a design draft for #3972: shift the cost of a stored attestation entry from the contract's balance onto the operator, without touching the node's deposit-less function-call key. The mechanism is a per-account grant counter — the operator prepays with a deposit-capable key in a separate transaction, submit_participant_info consumes one grant when it inserts a new entry, and clean_invalid_attestations hands the grant back when the entry goes away. The two protocol constraints the doc builds on (a function-call key cannot attach a deposit; report_data binds to env::signer_account_pk()) check out against nearcore semantics and crates/contract/src/tee/tee_state.rs:201, and the "pay and submit must be separate transactions" conclusion follows.

Changes:

  • New docs/design/operator-prepaid-attestation-storage.md: background on why nothing pays for attestation storage today, the grant-counter design, three-method contract API, fee sizing, charging rules, an onboarding sequence diagram, operator UX delta, decision/alternatives tables, security analysis, rollout, open questions and a test plan.

Reviewed changes

Per-file summary
File Description
docs/design/operator-prepaid-attestation-storage.md New design draft (240 lines): grant-counter funding model for attestation storage, with decisions, rejected alternatives, rollout and test plan.

Verified against the tree, and accurate: submit_participant_info is not #[payable] and its doc comment states storage is contract-funded (crates/contract/src/lib.rs:781); there is exactly one non-test stored_attestations.remove site (crates/contract/src/tee/tee_state.rs:471); vote_new_parameters does reject a proposal naming a participant without a valid attestation (crates/contract/src/lib.rs:929-948); the hourly re-attestation cadence matches crates/mpc-attestation/src/attestation.rs:25; the stale "will incur a cost (TBD, XXX NEAR)" text really is at docs/running-an-mpc-node-in-tdx-external-guide.md:1285; all six cross-doc anchors resolve; the fee arithmetic (604 → 0.00604, 734 → ~0.0073, 2.7×/1.4×) is internally consistent; and consuming at insert does leave #3991 unblocked, since no state needs to survive the failing callback (crates/contract/src/lib.rs:2340-2351). ParticipantInsertion::{NewlyInsertedParticipant, UpdatedExistingParticipant} already gives the insert site exactly the signal rules 1 and 2 need.

Findings

Blocking (must fix before merge):

  • docs/design/operator-prepaid-attestation-storage.md:116Rule 3 mints grants for entries that never consumed one. NodeAttestation (crates/contract/src/tee/tee_state.rs:87) has no "grant-backed" marker, so at removal time the contract cannot distinguish an entry that consumed a grant from one stored before the fee existed. Combined with :223 ("No grandfathering needed"), every pre-upgrade entry becomes a free grant the moment it is swept. Live nodes are unaffected (they re-attest under rule 1, so their entries never fail re-verification), but the abandoned pre-upgrade entries — precisely the ones rollout step 1 wants reclaimed, and now sweepable because fix(contract): make mock attestations cleanable via expiry #3785's stamp_expiry_on_legacy_mocks landed (crates/contract/src/v3_13_0_state.rs:122) — each convert into one permanently recyclable free entry slot. An attacker who front-runs the upgrade at today's ~7× amplification keeps that capacity forever, which breaks the entries held <= grants bought invariant asserted at :57. Please either mark grant-backed entries (a flag on the stored entry, or a snapshot of pre-upgrade TLS keys taken during migration) and credit only those, or state how many legacy entries convert to free grants and argue that bound is acceptable.

  • docs/design/operator-prepaid-attestation-storage.md:70, :213"Amplification drops below 1" is conditional on fee >= actual entry cost holding for the life of every outstanding grant, and grants never expire. Line 70 presents entry-denomination as a pure win ("never strands an operator whose existing grant is suddenly worth less than an entry") without naming who absorbs the shortfall: the contract. Because grants are unbounded in count, free of expiry, and recycled forever, they can be stockpiled at today's fee and redeemed after a storage-price rise or a layout change that outgrows the 2.7× headroom — at which point each redeemed grant is contract-subsidised storage and amplification climbs back above 1. Governance re-pricing does not help, since it only affects future grants (as line 70 itself says). The security analysis should state that exposure and bound it (cap outstanding grants, expire unused ones, or record the fee paid per grant), or at minimum qualify the "closes exactly" claim.

  • docs/design/operator-prepaid-attestation-storage.md:66 (and the goal at :41) — "Re-provisioned … indefinitely without another prepayment" does not hold, because a grant returns only once the old entry becomes invalid. A re-provisioned CVM generates fresh keys (docs/running-an-mpc-node-in-tdx-external-guide.md:1020; this is why step 3 re-adds the node account key), so the new submission is rule 2 — new TLS key, new map entry, one grant — while the old entry stays valid for up to DEFAULT_EXPIRATION_DURATION_SECONDS = 7 days (crates/mpc-attestation/src/attestation.rs:28). clean_invalid_attestations only removes entries that fail re-verification (crates/contract/src/lib.rs:1845-1864), and no method lets an operator surrender a still-valid entry, so a one-grant operator cannot re-provision without prepaying again or waiting a week. Either add a voluntary-release method (delete my entry for this TLS key, return the grant) to the API table and the operator UX section, or drop the claim and say re-provisioning needs a second grant just like migration does.

  • docs/design/operator-prepaid-attestation-storage.md:228The fee-sizing guidance contradicts the fee table. Line 104 correctly sizes the floor off charged storage (604 bytes mock / 599 dstack, the numbers pinned at crates/contract/src/lib.rs:8190-8191), but this bullet says to size the fee off "the mock worst case (450 borsh bytes), not Dstack's 445". 450 bytes is 0.0045 NEAR — below the 0.00604 floor for the entry alone — and the 450/445 pair appears nowhere in the repo; it looks like the serialised value without the key and IterableMap record overhead that the runtime does charge for. The "mock is the larger variant" point is right; please restate it with 604/599 or drop the byte figures, so nobody sizes the fee off the smaller number.

  • docs/design/operator-prepaid-attestation-storage.md:221, :217Rollout step 1 is already done. PR fix(contract): make mock attestations cleanable via expiry #3785 is merged and present on main (crates/contract/src/v3_13_0_state.rs:122,160), so "land it first or alongside" reads as a pending dependency that no longer exists. Related: reclaimability is not universal — per fix(contract): make mock attestations cleanable via expiry #3785's own out-of-scope note, TeeState::with_mocked_participant_attestations (crates/contract/src/lib.rs:1969) still stores non-expiring Mock::Valid sentinels, which never fail re-verification and so are never swept. Both statements need updating.

Non-blocking (nits, follow-ups, suggestions):

  • docs/design/operator-prepaid-attestation-storage.md:104, :196WORST_CASE_ENTRY_BYTES lives inside #[cfg(test)] mod tests (crates/contract/src/lib.rs:8133; module opens at 2818), so both mentions read as if it were a production constant. Fee-sizing alternative (a) would require promoting it — the test's own TODO(#3972) at crates/contract/src/lib.rs:8188 anticipates exactly this and is worth citing.
  • docs/design/operator-prepaid-attestation-storage.md:77available_attestation_grants is described as "what an operator checks to confirm a prepayment landed", but in steady state it returns 0 (available = bought − held, and line 118 deletes the row at zero), so a consumed prepayment is indistinguishable from none. Point operators at get_attestation / get_tee_accounts for the landed check, or reconsider not tracking the lifetime total.
  • docs/design/operator-prepaid-attestation-storage.md:116 — rule 3 adds a map write per removed entry — and a row insert when the counter row was deleted at zero — inside a gas-bounded sweep budgeted at DEFAULT_CLEAN_INVALID_ATTESTATIONS_TERA_GAS = 10 for RESHARE_CLEAN_INVALID_ATTESTATIONS_MAX_SCAN = 100 entries (crates/contract/src/config.rs:29, crates/contract/src/lib.rs:122). Worth a rollout line on re-checking that budget.
  • docs/design/operator-prepaid-attestation-storage.md:124 — the early check needs the owning-account comparison, not just key presence: store_verified_attestation rejects a TLS key registered to another account (crates/contract/src/tee/tee_state.rs:225-229), so a presence-only precondition would classify someone else's entry as "existing, no grant needed" and still reach verify_quote, weakening the stated fast-fail property.
  • docs/design/operator-prepaid-attestation-storage.md:76 — "Any remainder is kept" deviates from the contract's established deposit convention, require_deposit plus refund_to (crates/contract/src/lib.rs:135-157), which refunds the excess. Worth one sentence on why prepayment differs.
  • docs/design/operator-prepaid-attestation-storage.md:221 — rollout step 2 mentions only state migration, but making the fee votable also needs the ConfigExt DTO plumbing (crates/contract/src/dto_mapping.rs:483-487) and an updated borsh-schema snapshot (crates/contract/src/snapshots/mpc_contract__tests__mpc_contract_borsh_schema_has_not_changed.snap).

No embedded instructions to the reviewer, secrets, or injection attempts in the diff or PR body.

⚠️ Issues found

Resolves the open questions and the findings from the bot review on #4011.

Legacy entries: keep working with no grant (rule 1) and yield a grant when swept
(rule 3), both without extra code. The second is a deliberate grandfather -- 14
entries on mainnet, 31 on testnet -- but the exposure is precisely "entries
present at deploy time become permanent capacity", so Rollout now says to check
the count at deploy instead of assuming it. A count in the thousands would mean
the still-open drain was exploited before the release, and those entries should be
purged rather than granted.

Re-provisioning: a grant returns only once its entry expires, which takes 7 days,
so the earlier claim that an operator could prepay once and forget it was wrong.
Softened throughout: prepay for the nodes you run plus a spare, typically two.
An owner-callable release method would remove the need for the spare and is
recorded as an alternative, declined for now.

Fee is 0.02 NEAR (~2.7x the ~0.0073 floor). Multiple grants per call are
supported. The views are contract-only; operators read them with the NEAR CLI, so
nothing is added to the DTO/ABI surface.

Also: state that all fee figures are charged bytes rather than borsh sizes, which
was mixing 604 and 450; note the early check rejects a TLS key owned by another
account so it does not burn verification gas; and record the price risk on
outstanding grants as an accepted bounded residual.
- Rule 3 costs gas in a gas-bounded sweep. Every removed entry adds a grants-map
  write, and that write is a row insert rather than an update whenever the owner's
  row was deleted at zero. clean_invalid_attestations_tera_gas and the
  RESHARE_CLEAN_INVALID_ATTESTATIONS_MAX_SCAN default were sized for removals
  alone and need re-validating, with a budget guard of the kind #3936 adds. This
  was missed entirely in the earlier drafts.

- Rule 1 turns on ownership, not key presence. Say so explicitly: an early check
  that only tests whether the TLS key exists would classify a submission for
  someone else's key as "existing entry, no grant needed", pass it, verify it, and
  only then fail on TlsKeyOwnedByOtherAccount.

- WORST_CASE_ENTRY_BYTES lives in a #[cfg(test)] module and the fee no longer
  derives from it, so stop referencing it as though it were a production constant;
  name the test that pins 604 instead.

- available_attestation_grants returns 0 both for "never prepaid" and for
  "prepaid, now backing an entry", since the row is deleted at zero. Explain how
  to disambiguate with get_tee_accounts.
netrome
netrome previously approved these changes Jul 30, 2026

@netrome netrome left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc feels a bit verbose, and I think it could be improved if we slimmed it down and put more emphasis on the core model earlier on in the doc, combined with some sequence diagrams.

But since this is transient, and we have alignment on the core idea I don't think it's critical to polish this doc.

Marten approved but asked for it to be slimmed; Reynaldo has not been able to read
it because of the length, which makes trimming the blocker on his review rather
than a nicety.

259 lines -> 160. No decisions changed. The cuts are duplication and background:

- Goals and Non-goals removed; the opening paragraph and Alternatives already
  carried both.
- Background compressed: the nearcore snippet and the FunctionCallPermission /
  meta-transaction detail become one line each, keeping only why payment and
  submission cannot be done by the same party.
- "No NEAR is ever refunded" appeared three times, the 7-day spare-grant point
  three times, and the legacy grandfather in three sections. Each now has one
  home, cross-referenced.
- Decisions and Alternatives tables keep every row but lose the prose; each is now
  a clause rather than a paragraph.
- Operator UX drops the seven-row status table, which restated the guide's own
  contents, and says instead that only one step is added and nothing else changes.
- Rollout renamed to Implementation notes, since that is what it is now that the
  design is settled.
…ependency

Two facts from the full bot review that the earlier summary did not surface:

- #3785 is merged (2026-07-29), so "land it first or alongside" described a
  dependency that no longer exists. Replaced with what actually remains: mock
  entries are sweepable now, but TeeState::with_mocked_participant_attestations
  still stores bare non-expiring Mock::Valid sentinels at init, which never fail
  re-verification and so are never swept or granted.

- The legacy grandfather does not work the way we assumed when deciding to accept
  it. A live node re-attests under rule 1, so its entry never fails
  re-verification, is never swept, and yields no grant -- operators currently
  running nodes get nothing and need nothing. Only *abandoned* entries convert to
  grants. The number is still negligible today (14 mainnet, 31 testnet, nearly all
  live) and the deploy-time count check still bounds it, but the rationale is
  "abandoned entries are rare", not "it rewards our existing operators".

Also: say why prepay keeps the remainder instead of following the contract's
require_deposit + refund_to convention, and note that making the fee votable needs
the ConfigExt DTO plumbing and a borsh-schema snapshot, not just the migration.
@barakeinav1

Copy link
Copy Markdown
Contributor Author

Thanks — went through all ten. Nine addressed, one accepted as a bounded residual. Also slimmed the doc from 259 to 162 lines, per @netrome and @gilcu3 in Slack.

Rule 3 mints grants for entries that never consumed one. … every pre-upgrade entry becomes a free grant the moment it is swept.

Accepted deliberately rather than fixed, but your analysis corrected our reasoning for accepting it, so the doc now says something different from what we decided in the meeting.

We had reasoned "it's a small bounded gift to the operators already running nodes". That was wrong, and your note is why: a live node re-attests under rule 1, so its entry never fails re-verification, is never swept, and yields no grant. Live operators get nothing — and need nothing. Only abandoned entries convert, which is the opposite of the population we thought we were rewarding.

The bound still holds: 14 entries on mainnet and 31 on testnet, nearly all live, so the abandoned set is ~0 today. What changed is that the rationale is now "abandoned entries are rare", not "this rewards our operators" — and because that is only true at deploy time, the doc makes it a release-checklist item: check the stored-entry count before shipping, and purge rather than grant if it is in the thousands, which would mean the still-open drain was exploited first. Adding a grant-backed marker would cost ~110 bytes per entry on the fee and a second map, which we judged not worth it at this scale.

"Amplification drops below 1" is conditional on fee >= actual entry cost holding for the life of every outstanding grant, and grants never expire.

Agreed and now stated rather than implied. The security section records it as an accepted residual: exposure is bounded by outstanding grants, the contract stores counts rather than purchase prices so it cannot even identify which grants are underwater, governance can re-price only future grants, and a future update can address it if the gap becomes material. We took the operator-certainty side of that trade knowingly.

"Re-provisioned … indefinitely without another prepayment" does not hold, because a grant returns only once the old entry becomes invalid.

Correct, and the claim is dropped. A fresh CVM means a new TLS key, so the new submission is rule 2 while the old entry stays valid for up to DEFAULT_EXPIRATION_DURATION_SECONDS. The doc now says an operator prepays for the nodes they run plus a spare — typically two, the live node and its migration target — and a voluntary-release method is recorded under Alternatives as declined for now, since one spare grant is cheaper than another entry point that has to refuse to evict a live participant. Worth revisiting if the 7-day wait proves painful.

The fee-sizing guidance contradicts the fee table. … 450 bytes is 0.0045 NEAR — below the 0.00604 floor for the entry alone

Fixed. All figures are now explicitly charged bytes, and the mock-is-larger point is restated as 604 vs 599 with the borsh pair removed from the sizing guidance entirely.

Rollout step 1 is already done. PR #3785 is merged and present on main

Fixed — this one the earlier summary I was working from had omitted, so thanks for the specificity. Step 1 now records that mock entries are already sweepable and that reclaimability is still not universal: with_mocked_participant_attestations stores bare non-expiring Mock::Valid sentinels at init, which never fail re-verification and so are never swept or granted.

Non-blocking

WORST_CASE_ENTRY_BYTES lives inside #[cfg(test)] mod tests … both mentions read as if it were a production constant

Fixed. The constant is no longer cited; the doc names the test that pins 604 and notes the fee is a Config value, so nothing in the contract reads it. Alternative (a) would have needed the promotion you describe, which is one more reason we did not take it.

available_attestation_grants … in steady state it returns 0 … a consumed prepayment is indistinguishable from none

Fixed: the doc now says 0 is ambiguous and to disambiguate with get_tee_accounts — no entry means prepay, entry present is the normal steady state.

rule 3 adds a map write per removed entry — and a row insert when the counter row was deleted at zero — inside a gas-bounded sweep

Good catch, this was missed entirely. Now an implementation note: re-validate clean_invalid_attestations_tera_gas and RESHARE_CLEAN_INVALID_ATTESTATIONS_MAX_SCAN against the heavier per-entry cost, with a guard test of the kind #3936 adds, plus a worst-case test (every scanned entry removed, every owner's row absent so each write is an insert).

the early check needs the owning-account comparison, not just key presence

Fixed. Rule 1 now says explicitly that it keys on ownership rather than presence, and why: a presence-only precondition would classify someone else's entry as "existing, no grant needed" and still reach verify_quote.

"Any remainder is kept" deviates from the contract's established deposit convention

Fixed — one sentence on why: a grant is a discrete unit, so the leftover is at most one fee short of the next grant, and a transfer path for sub-0.02 NEAR dust is not worth it.

rollout step 2 mentions only state migration, but making the fee votable also needs the ConfigExt DTO plumbing … and an updated borsh-schema snapshot

Fixed, both named.

Also updated #3972's acceptance criteria, which had drifted from the agreed model — it still required refunding excess and still claimed an operator could pay the first submission on the node's behalf, which the report_data binding makes impossible.

- Status line removed; the design is settled, not a draft for review.
- Testing removed outright: the test plan belongs in the implementation PR.
- Implementation notes removed, but two of its items were costs of design
  decisions rather than implementation chores, so they moved next to the decisions
  that cause them instead of disappearing: rule 3 now carries its own sweep
  gas-budget caveat, and the votable-Config row in Decisions carries the state
  migration, ConfigExt plumbing and snapshot regeneration. The non-universal
  sweepability caveat moved to the Security residual about swept entries, which is
  what it qualifies.

Dropped entirely: the #3785 dependency note (merged, so there is nothing to
sequence) and the runbook-coordination reminder (Operator UX already says
operators must prepay first).

Also repaired the API table, which an earlier edit had split by inserting a
paragraph between its rows.

259 -> 144 lines across this and the previous trim.
prepay_attestation_storage(account_id, grants) now requires exactly fee x grants
and rejects anything else. That is clearer about intent than floor(attached / fee)
and it removes the remainder question altogether -- there is nothing left over to
keep or refund, so the justification for keeping it goes too, along with the
Copilot finding it was answering. The operator reads attestation_storage_fee()
first and attaches the exact multiple; one call now covers every node they run.

Also, per review: "a count (not an amount)"; drop the line about views being
contract-only; drop the note that 604 is pinned by a test; drop the stale-guide
note, which belongs in the implementation PR; and shorten the pre-upgrade-grant
residual to the decision and the deploy-time check.
netrome
netrome previously approved these changes Jul 30, 2026

@netrome netrome left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating


### Fee

0.02 NEAR, about 2.7× the floor. Figures are **charged** bytes — key and record overhead included — not borsh sizes:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation note: I think this should be configurable on the contract and not hard-coded.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and that's the design already — it's a governance-votable Config field, not a constant. The doc said so eight lines below your comment (and again in the Decisions table), which is too far from the number to count.

Moved it onto the number itself: "0.02 NEAR — a governance-votable Config field, not a constant — about 2.7× the floor."

@barakeinav1

Copy link
Copy Markdown
Contributor Author

Addressed all open review comments in db9badf — replies are on each thread. Two are design changes, not wording:

Prepay moves to node-account creation (@gilcu3, flow + operator UX). We were both anchoring it too late. prepay_attestation_storage needs only the account id, and the operator holds that account's full-access key at Create a NEAR Account for Your Node — before the CVM is started, before the node key is retrieved or added. So the node never starts into an ungranted state, and the sync-timing question is moot.

Existing entries are grandfathered by the migration (@gilcu3 on the residual). One grant credited per account already holding an entry, so rule 3 returns grants that were genuinely issued. Removes an accepted residual and the rule-3 asymmetry, without the per-entry marker I'd rejected on cost. Same 14/31 free slots, issued explicitly up front; the deploy-time count check stays.

Rest: fee is now labelled a votable Config field where the number appears (@netrome — it already was, just stated too far down); the 604/599 figures now cite the env::storage_usage() measurement behind them rather than asserting them (@gilcu3 — and my "0.02 gives buffer" reply was the wrong defence); plainer wording for the grant-semantics row and the price-risk residual.

This PR closes #3972, which is now the design issue, so a TODO(#3972) left in the
tree fails the closed-issue check. The deposit sizing it refers to is
implementation work, tracked in #4015.
@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

PR title type suggestion: This PR includes changes to source code files (crates/contract/src/lib.rs), not just documentation. The type prefix should be feat: instead of docs:.

Suggested title: feat: add operator-prepaid attestation storage

…on crediting

db9badf had the state migration credit one grant per existing entry so that
rule 3 only ever returns grants that were genuinely issued. Reverting that: it is
more code, not less -- an iteration plus a per-account write inside
From<MpcContract> -- and it buys only the tidiness of the invariant reading
uniformly, which is a documentation nicety rather than a functional gain.

The intended behaviour is that nothing happens for existing entries. An entry that
predates the fee already holds a slot and no grant was ever bought for it, so it is
in effect a grant already spent. Re-attestation is free under rule 1, so those
operators need no grant and no action, and the contract needs no migration step, no
per-entry marker and no second map.

Moved out of Security into its own "Existing nodes" section and written plainly, so
it reads as a decision taken on purpose rather than a residual risk discovered
later. The deploy-time count check stays. Security keeps the one real residual, the
price risk on sold grants.
Four paragraphs to three, same three facts: nothing is owed to already-attested
nodes, nothing in the contract handles them specially, and the one consequence is
accepted on purpose.
@barakeinav1
barakeinav1 requested review from gilcu3 and netrome July 30, 2026 17:09
@barakeinav1
barakeinav1 enabled auto-merge July 30, 2026 17:09
gilcu3
gilcu3 previously approved these changes Jul 31, 2026

@gilcu3 gilcu3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates

Replied to some comments. We still don't fully agree on some of them, but this is approved either way as none of them are blockers to me

Comment on lines -8188 to 8197
/// TODO(#3972): the flat onboarding deposit will be derived from these sizes too.
/// TODO(#4015): the prepaid-storage fee is sized from these numbers.
#[rstest]
#[case::dstack(599, worst_case_dstack_attestation())]
#[case::mock(604, worst_case_mock_attestation())]
fn stored_attestation_entry__should_have_the_pinned_size(
#[case] expected_bytes: u64,
#[case] verified_attestation: VerifiedAttestation,
) {
// Given / When
let bytes_stored = measure_stored_entry_bytes(verified_attestation);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reference, but this does not measure real deposit costs, as it only inserts one attestation into an empty state

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see here some measurement results #4011 (comment)

Comment on lines +137 to +139
Two accepted residuals:

- **Pre-upgrade entries yield a grant when swept**, since rule 3 cannot distinguish them from paid ones. Accepted: there are 14 such entries on mainnet and 31 on testnet, so the free capacity is negligible (and init-time mock sentinels never expire, so they are never swept and never yield one). Check the count at deploy though — a count in the thousands would mean the still-open drain was exploited first, and those entries should be purged rather than granted.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but with_mocked_participant_attestations still inserts non-expiring MockAttestation::Valid sentinels at init (crates/contract/src/tee/tee_state.rs:136-142), so those are never swept. Grandfathering handles them cleanly: the sentinel's account gets its grant, and a real submission for the same TLS key is rule 1, which consumes nothing.

oh, if that is true, why didn't we close that gap in #3785 ?

Comment on lines +58 to +62
| Component | Charged bytes | Cost |
|---|---|---|
| Worst-case entry: a `Mock` one at 604 (`Dstack` is 599) | 604 | 0.00604 NEAR |
| Grants-map row | ~130 | ~0.0013 NEAR |
| **Floor** | **~734** | **~0.0073 NEAR** |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This benchmark is still not so accurate for what we need. The underlying data structure is a hash map, so its total size "per entry" is not as easy to compute. To really figure this out we would need to insert many attestations and then get the average or maximum of the cost per attestation. Therefore, we still don't have proper numbers, but we could do it as part of the PR following this design. No need to mention an amount here, except as a rough estimate

|---|---|---|
| `prepay_attestation_storage(account_id, grants)` | `#[payable]` | Adds `grants` to `account_id`. Requires an attached deposit of exactly `fee × grants` and rejects anything else, so there is no remainder to keep or refund. Permissionless — anyone may prepay for any account. |
| `available_attestation_grants(account_id) -> u32` | view | Grants available. |
| `attestation_storage_fee() -> NearToken` | view | Current fee. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fee is votable, so it changes without a release

Right, we are making it votable :) Probably we didn't need that though, I don't see operators ever changing that, but that's fine. But again, if the operator can already get the value from config() then we don't need this convenience. Extracting the value from the config blob is trivial, and we do not expect any operator to do this automatically anyway, as this is meant to be a manual step after all, specially for a value that would almost never change.

@claude

claude Bot commented Aug 1, 2026

Copy link
Copy Markdown

PR title type suggestion: This PR changes only configuration files, font assets, and build artifacts—not documentation. The type prefix should probably be chore: instead of docs:.

Suggested title: chore: add fonts and update VS Code config (or similar, depending on the actual intent)

@andrei-near
andrei-near force-pushed the 3972-operator-prepaid-attestation-storage branch from 93c9ab5 to c46da65 Compare August 1, 2026 16:02
@claude

claude Bot commented Aug 1, 2026

Copy link
Copy Markdown

PR title type suggestion: This PR modifies both source code (crates/contract/src/lib.rs) and documentation. Since source code changes are included, the type prefix should reflect the primary code change — likely feat: instead of docs:.

Suggested title: feat: add operator-prepaid attestation storage (or similar, depending on what the contract change implements)

@barakeinav1

Copy link
Copy Markdown
Contributor Author

The underlying data structure is a hash map, so its total size "per entry" is not as easy to compute. To really figure this out we would need to insert many attestations and then get the average or maximum of the cost per attestation.

Ran that experiment — 200 sequential inserts into stored_attestations, taking the env::storage_usage() delta of each one individually:

first=232  2nd=232  50th=233  100th=233  200th=234
distinct deltas across all 200 = [232, 233, 234]

Per-entry cost is constant in map size. The 2-byte drift is my probe's own fault: I named the accounts op0.nearop199.near, so the account id grew by two characters between the first insert and the last. Nothing to do with occupancy.

The reason is that near_sdk::store::IterableMap is not a bucketed hash map. It is a LookupMap (one trie record per entry, keyed by prefix + serialised key) plus a Vector of keys (one record per element). NEAR's storage accounting charges per record — key bytes + value bytes + a fixed 40-byte per-record overhead — and never for trie internal nodes. So the marginal cost of the Nth insert does not depend on N, and averaging over many inserts returns the same number as measuring one.

That means measure_stored_entry_bytes inserting into empty state is a valid measurement rather than a best case, and 604 mock / 599 dstack stand. Happy to add this probe as a permanent test in the implementation PR if you would rather it were pinned than argued — it is ~20 lines.

Two caveats I do want to keep, and the doc says both:

  • The ~130 bytes for the grants-map row is an estimate, not a measurement. It is a much smaller term than the entry and the fee's headroom covers it, but it has not been measured the way 604 has.
  • The fee is deliberately ~2.7× the ~734-byte floor precisely so that neither figure has to be exact.

@gilcu3: config() already returns the fee, and reading it is a manual step on a
value that almost never changes, so a dedicated view may not earn a permanent place
in the API. Marked optional and left out of the implementation until we decide --
adding it later is cheaper than removing it. The operator UX now reads
attestation_storage_fee_millinear from config() instead.
barakeinav1 added a commit that referenced this pull request Aug 3, 2026
@gilcu3 on #4011: config() already returns attestation_storage_fee_millinear, and
an operator reads it once by hand on a value that almost never changes, so a
dedicated view does not earn a permanent place in the API or the ABI.

Both test harnesses now read the fee the way an operator does -- the in-process one
through config(), the sandbox one through the config view -- so nothing depends on
a contract method existing purely for convenience. The internal helper stays, since
prepay_attestation_storage needs the value to check the deposit.

Left a TODO(#4015) noting the view can be added later if operators ask; adding is
cheaper than removing.
@claude

claude Bot commented Aug 3, 2026

Copy link
Copy Markdown

PR title type suggestion: This PR adds new contract implementation code in addition to design documentation. The type prefix should be feat: instead of docs:.

Suggested title: feat: operator-prepaid attestation storage

@barakeinav1
barakeinav1 requested a review from gilcu3 August 3, 2026 10:58
@claude

claude Bot commented Aug 3, 2026

Copy link
Copy Markdown

PR title type suggestion: This PR changes source code (crates/contract/src/lib.rs) in addition to documentation, so the type prefix should reflect the source code intent rather than docs:. Consider using feat: if this adds new functionality or refactor: if restructuring existing code.

Suggested format: feat(design): draft operator-prepaid attestation storage (or appropriate type based on the contract changes)

@barakeinav1 barakeinav1 changed the title docs(design): draft operator-prepaid attestation storage docs(design): operator-prepaid attestation storage Aug 3, 2026
@claude

claude Bot commented Aug 3, 2026

Copy link
Copy Markdown

PR title type suggestion: This PR adds a new source code file (8K+ lines in crates/contract/src/lib.rs), not just documentation. Since the primary change is a feature implementation, the type prefix should be feat: instead of docs:.

Suggested title: feat: operator-prepaid attestation storage

@netrome netrome left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!


### Charging rules

Evaluated read-only at the top of `submit_participant_info` — before any verification, so an ungranted or unauthorised call never reaches the `Mock` checks or a `verify_quote` round trip — and applied at insert.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Don't see the point in distinguishing the different verifications we have here

Suggested change
Evaluated read-only at the top of `submit_participant_info` — before any verification, so an ungranted or unauthorised call never reaches the `Mock` checks or a `verify_quote` round trip — and applied at insert.
Evaluated read-only at the top of `submit_participant_info` — before any verification, so an ungranted or unauthorised call never reaches attestation verification checks.


Accepted deliberately: if such an entry is later swept, rule 3 hands its owner a grant they never bought. Negligible at 14 entries on mainnet and 31 on testnet, but check the count before deploying — thousands would mean the drain was exploited first, and those entries should be purged rather than left to become grants.

## Operator UX

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels a bit strange but sure, it's worth keeping as a reminder to update the operator guide - although we should always remember to do this for operator-facing changes.

@barakeinav1
barakeinav1 added this pull request to the merge queue Aug 3, 2026
Merged via the queue into main with commit f13945f Aug 3, 2026
16 of 17 checks passed
@barakeinav1
barakeinav1 deleted the 3972-operator-prepaid-attestation-storage branch August 3, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

design: funding model for attestation storage [Docs-Missing] Document cost in NEAR tokens for calling submit_participant_info

4 participants