Skip to content

[core] Handle query auth wrappers in split consumers#8711

Draft
discivigour wants to merge 31 commits into
apache:masterfrom
discivigour:bug/j/queryAuthSplit
Draft

[core] Handle query auth wrappers in split consumers#8711
discivigour wants to merge 31 commits into
apache:masterfrom
discivigour:bug/j/queryAuthSplit

Conversation

@discivigour

@discivigour discivigour commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fix failures and incorrect reader routing when query authorization wraps planned splits in QueryAuthSplit while specialized split consumers assume they receive raw DataSplit instances.

The affected paths include:

  • fallback branch scans, where fallback metadata and query-auth wrappers can be nested;
  • cross-partition index bootstrap, which must build the index from unfiltered and unmasked primary keys;
  • chain table batch and streaming scans, which replace DataSplit with ChainSplit;
  • data evolution scans, which replace DataSplit with IndexedSplit; and
  • the top-level chain table reader, which must route query-auth-wrapped chain and delta splits to the chain-aware reader.

Changes

  • Add one-layer QueryAuthSplit.unwrap, unwrapDataSplit, and retainAuth helpers.
  • Make fallback planning accept generic Split values. Reader dispatch removes only the fallback wrapper and preserves the query-auth wrapper for row filtering and column masking.
  • Make IndexBootstrap explicitly disable query auth on its internal latest-snapshot read. This ensures the index is built from the original primary keys instead of query-filtered or column-masked values.
  • Preserve query authorization when chain table batch and streaming scans transform DataSplit into ChainSplit, including lightweight and chain-table.streaming.merge-snapshot starting plans.
  • Route QueryAuthSplit(ChainSplit) and QueryAuthSplit(DataSplit) through the chain-aware reader while passing the original wrapper to retain authorization context.
  • Preserve query authorization when data evolution scans transform DataSplit into IndexedSplit.
  • Add regression coverage for fallback planning/serialization/reading, index bootstrap, chain table batch and streaming scans, chain reader routing, data evolution, and split helper serialization.

Tests

mvn -pl paimon-core -am -Pfast-build \
  -DfailIfNoTests=false \
  -DwildcardSuites=none \
  -Dtest=ChainTableFileStoreTableTest,ChainGroupReadTableTest,FallbackReadFileStoreTableTest,DataEvolutionBatchScanTest,IndexBootstrapTest,QueryAuthSplitTest \
  test

Result: 35 tests passed.

Final validation:

mvn -pl paimon-core -DskipTests validate

Checkstyle, Spotless, and Maven Enforcer passed.

@discivigour
discivigour force-pushed the bug/j/queryAuthSplit branch from 96d4d8b to e756304 Compare July 20, 2026 04:22

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the broad wrapper coverage. I found three blocking wrapper compositions in the current master merge tree: nested query-auth wrappers in lateral PK vector search, fallback wrappers around query-auth splits in Flink dynamic filtering, and score-reader type erasure after authorization filtering. I reproduced the Spark cases with Spark 3 on JDK 11 and the fallback case with a focused Flink 1 regression. The existing focused tests pass, but the positive/composed-path regressions fail as described inline.

@leaves12138
leaves12138 dismissed their stale review July 20, 2026 10:20

Submitted without the requester explicitly asking for GitHub comments. Withdrawing this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants