Fenestra

Fenestra implements the Monstera core snapshot stream format. The name comes from the fenestrations (the holes) of a monstera leaf.

Fenestra was crated for application cores with multiple logical tabels (maps, arrays, etc) that do not support any other snapshotting mechanism natively which would satisfy Monstera shard splitting requirements: a child core should be able to restore from its parent's snapshot and keep only relevant records.

A snapshot is a flat, single-pass, write-once stream of frames. Rows are logically addressed as (table id, canonical key): table ids are application-wide stable UTF-8 names, and keys carry no storage prefixes, which makes snapshots portable between cores of one application.

The format never requires the producer to know record counts upfront (a storage range scan can stream rows directly into a Writer) and never requires the consumer to look ahead (a Reader yields frames in one pass and fails with ErrTruncated if the stream ends before the END frame).

Encoding

A snapshot is a flat sequence of frames:

snapshot := header frame* END
frame    := TABLE | RECORD | META

A TABLE frame declares the current table, and the RECORD frames that follow belong to it until the next TABLE frame. A table may be declared more than once (its record sets concatenate). Tables may appear in any order. A table with no records may simply be omitted — Restore semantics are whole-state replacement (the consumer clears all of its state first), so absence means empty.

All integers are unsigned LEB128 varints (encoding/binary Uvarint), max 10 bytes. All byte strings are length-prefixed with a varint. Frame tags are single bytes.

header   := magic version
magic    := "MSNP"                    (4 bytes: 0x4D 0x53 0x4E 0x50)
version  := uvarint                   (today 1)

TABLE    := 0x01 len name flags
name     := len bytes of UTF-8        (the table id; len > 0)
flags    := uvarint                   (bit 0: records of this section are sorted
                                       ascending by key; all other bits reserved,
                                       must be 0 in version 1)

RECORD   := 0x02 klen key vlen value  (klen ≥ 0, vlen ≥ 0; belongs to the current
                                       table — a RECORD before any TABLE is malformed)

META     := 0x03 klen key vlen value  (informational key/value annotation, e.g.
                                       "application", "shard_id", "raft_index";
                                       consumers may ignore every META frame;
                                       restore correctness must never depend on one)

END      := 0x00                      (mandatory; the stream ends here. A stream
                                       that stops without END is truncated and the
                                       consumer must fail the restore)

If the same (table id, key) appears twice, the later record wins. Producers should not emit duplicates; consumers are not required to detect them.

Zero-length keys and values are legal (a singleton row can live at the empty key; a set membership row can carry an empty value).

Consumers must enforce sanity limits on the varint lengths before allocating (defaults: key ≤ 1 MiB, value ≤ 256 MiB, table id ≤ 4 KiB; configurable). A corrupt length must fail the restore, not attempt the allocation.

Because replicas of one shard produce snapshots independently, streams are semantically equal but need not be byte-identical. The first reason is replica-specific META such as timestamps and replica ids. Another reason is a non-deterministic map iteration order. Nothing in the system compares snapshot bytes.

Example

A core with two tables — "kv" (two rows, sorted) and "idx" (one row) — annotated with the producing shard:

4D 53 4E 50            magic "MSNP"
01                     version 1
03  0B "application"   META
    07 "example"
01  02 "kv"            TABLE "kv"
    01                 flags: sorted
02  03 "foo"           RECORD key="foo"
    05 "hello"                value="hello"
02  03 "zap"           RECORD key="zap"
    05 "world"                value="world"
01  03 "idx"           TABLE "idx"
    00                 flags: none
02  02 [03 E8]         RECORD key=(0x03E8="1000" big-endian)
    03 "foo"                  value="foo"
00                     END

Snapshot contract

Stream tables in any order. Within a table, iterate the range and emit each row with any storage prefixes stripped (physical key → canonical key is a fixed, per-table transformation). The producer must emit every row the core owns and nothing else. The format never requires the producer to hold more than one row at a time.

Restore contract

First, clear all own state. Read frames in one pass. For each RECORD, resolve the current table id to the local table, then write the row at the local physical location for (table id, key).

Fail on: bad magic, unknown version, unknown frame tag, RECORD before TABLE, limit violations, unknown table id, or missing END.

An unknown table id is an error (not a skip): within one application every core version knows the full table list, and silently dropping a table could cause data loss.

Important: Key/Value buffers are reused between Next calls. Consumers that need to retain them should copy explicitly.