Skip to content

Commit 71c87f7

Browse files
committed
Clarify chain table streaming read merge-snapshot and changelog-producer restrictions
1 parent 5e9afd7 commit 71c87f7

1 file changed

Lines changed: 19 additions & 19 deletions

File tree

docs/docs/primary-key-table/chain-table.mdx

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -130,7 +130,6 @@ Notice that:
130130
- Chain table is only supported for primary key table, which means you should define `bucket` and `bucket-key` for the table.
131131
- Chain table should ensure that the schema of each branch is consistent.
132132
- Deletion vector is not supported for chain table.
133-
- The `changelog-producer` option must be `none` (default) or `input`; `lookup` and `full-compaction` are not supported for chain tables.
134133
- The delta branch must use the `DEDUPLICATE` merge engine (default) if you plan
135134
to use streaming read or lookup join. Other merge engine types are not supported
136135
on the delta branch for these incremental read paths. Batch read is not affected.
@@ -226,8 +225,8 @@ Chain tables support Flink streaming read. A streaming read job operates in two
226225
1. **Full load phase**: By default it produces a lightweight result by reading the latest
227226
snapshot partition (per group) and delta partitions that come after it. Older snapshot
228227
partitions are excluded. You can enable `chain-table.streaming.merge-snapshot` to perform
229-
anchor-based chain merging in Phase 1, allowing cross-branch deletes and updates to be
230-
visible in the starting snapshot.
228+
anchor-based chain merging in this phase, allowing cross-branch `DELETE` records to be
229+
resolved together with the snapshot data.
231230
2. **Incremental phase**: Continuously reads new commits from the delta branch as they arrive.
232231

233232
### Write-Side Requirements
@@ -254,32 +253,27 @@ SET 'execution.runtime-mode' = 'streaming';
254253
INSERT INTO downstream_sink SELECT * FROM default.t;
255254
```
256255

257-
### Merge Snapshot in Starting Phase
256+
### Merge Snapshot in Full Load Phase
258257

259-
By default, the full load phase is lightweight: for each group it reads the latest
260-
snapshot partition and later delta partitions as separate splits. Anchor-based chain
261-
merging is skipped to keep Phase 1 fast and avoid split explosion in jobs with many
262-
partitions.
258+
By default, the full-load phase is lightweight: for each group it reads the latest snapshot and later
259+
delta partitions as separate splits. This is fast but cross-branch deletes are invisible — the `DELETE`
260+
records in the delta branch cannot be deleted in the snapshot branch.
263261

264-
If the delta branch contains deletes or updates that must be merged with the latest
265-
snapshot anchor — for example, a later delta partition deletes a key that exists in the
266-
latest snapshot partition — you can enable the merge mode:
262+
If you need a fully reconciled starting snapshot, enable merge mode:
267263

268264
```sql
269265
ALTER TABLE default.t SET (
270266
'chain-table.streaming.merge-snapshot' = 'true'
271267
);
272268
```
273269

274-
When enabled, the starting phase merges the latest snapshot partition per group with
275-
delta partitions whose chain key is strictly greater than the snapshot chain key. This
276-
produces a correct starting snapshot that reflects cross-branch deletes and updates,
277-
at the cost of a heavier Phase 1.
270+
With merge mode enabled, the full-load phase merges the latest snapshot partition per group with
271+
delta partitions whose chain key is strictly greater than the snapshot's, so cross-branch
272+
deletes are correctly resolved. The trade-off is a heavier startup scan.
273+
274+
To reduce the overhead, run `CALL sys.compact_chain_table(...)` periodically.
275+
After compaction, only the delta changes that arrived after compaction need to be merged.
278276

279-
To reduce this overhead, you can periodically run `compact_chain_table` to merge delta
280-
partitions into the snapshot branch. After compaction, the snapshot branch contains the
281-
merged state, so the streaming starting phase can read snapshot partitions directly
282-
without expensive cross-branch merging.
283277

284278
### Limitations
285279

@@ -298,6 +292,12 @@ without expensive cross-branch merging.
298292
specific partition, use batch mode instead.
299293
- The delta branch must use the `DEDUPLICATE` merge engine (default). Other merge engine
300294
types are not supported.
295+
- The `changelog-producer` option must be `none` (default) or `input`; `lookup` and `full-compaction`
296+
are not supported for chain tables.
297+
- When `changelog-producer` is `none`, Flink's operator normalizes records by the full
298+
primary key including the chain partition. The records of `-D`/`-U` in a different chain partition
299+
than the original records of `+I` will be dropped. Use `input` if downstream must receive cross-partition
300+
changelog records.
301301

302302
## Lookup Join
303303

0 commit comments

Comments
 (0)