[core][spark] Support catalog-managed Format Table reads and writes

[sundapeng]

Relationship

First part of splitting #8713 into independently reviewable pieces (this PR is the
engine-independent foundation; the Spark DDL and administration parts follow in separate
PRs). The content is identical to the corresponding files of #8713.

Purpose

Follow-up of #8707 (catalog-managed Format Table partitions, REST contract + server side).
This PR makes the engine-independent client side actually use the managed partition
metadata: scans plan from the catalog instead of walking the directory tree, and batch
writes register the partitions they touched as part of the commit.

Main changes

• Catalog#supportsManagedFormatTablePartitions capability (default false; REST catalog
  returns true), mirroring the existing capability-method pattern.
• ManagedFormatTableScan: partitions come from paged catalog listing with a
  partition-name prefix pattern pushed down; no filesystem fallback. A registered
  partition whose directory is missing reads as empty with a WARN (Hive semantics —
  an ADD PARTITION before the first insert is a legitimate empty partition), while a
  catalog with no partitions but a non-empty table directory warns that a metadata sync
  is needed. Catalog metadata is validated defensively (corrupt/duplicate specs).
• FormatTableCommit: batch writes register touched partitions with the idempotent
  create contract, in batches of the catalog page size (one backfill commit can touch
  tens of thousands of partitions). Registration failure fails the job; data files are
  never silently dropped.
• Create validation (REST catalog only): the managed option is rejected on external
  Format Tables and on the engine implementation at CREATE TABLE time, with catalog
  table-default.* options participating in the effective combination. Other catalogs
  keep treating the option as inert. Any table carrying the option where it cannot be
  honored (other catalogs, external tables, later ALTER ... SET) still loads,
  downgraded to unmanaged with a WARN, and can be repaired with
  ALTER TABLE ... RESET/UNSET — there is deliberately no ALTER-time validation
  machinery in this first version.
• PartitionPathUtils hardening shared by the scan/discovery paths.

Behavior notes reviewers should be aware of

1. Tightened shared utils: generatePartitionPathUtil now rejects null/empty
   partition values (they cannot form a path component), and value-only spec extraction
   unescapes directory names so specs round-trip with what writers register. Existing
   tables with %xx-escaped value-only directories will surface unescaped values.
2. In-process listing cache: partition listings are cached for up to 30s per JVM
   (invalidated on same-process mutations, including attempted-but-ambiguous ones).
   Cross-JVM readers may observe metadata up to 30s stale; same-process reads are
   immediately consistent.
3. Reads-empty drift semantics (vs failing the scan) was chosen deliberately: an
   empty registered partition is indistinguishable from an out-of-band directory
   deletion, and Hive treats both as empty. Repair paths stay fail-closed (see the
   administration part).

Tests

Core: managed scan (19), commit registration (7), catalog validation/fallback (core +
FileSystem/Jdbc/Hive/REST catalog suites, 354 tests in paimon-core plus 53 in
HiveCatalogTest), compatibility test pinning the Catalog binary surface, path-utils
tests. Spark write glue: batch write test.

[sundapeng]

Closing in favour of a re-split of this work.

The implementation changed enough since this split that updating the branch in
place would replace every line and leave the review comments pointing at code
that no longer exists:

• the serializable provider with its 30s listing cache and JVM-wide generation
  counter is gone, replaced by a partition manager that creates one catalog per
  operation and closes it;
• the managed scan subclass is gone, folded into FormatTableScan;
• the two procedures are gone; MSCK REPAIR TABLE is the only entry point;
• the feature no longer reuses metastore.partitioned-table. It has its own
  option, format-table.partition-source = filesystem | rest, so a
  catalog-level default set for Paimon tables cannot turn it on for a Format
  Table by accident.

The review comments here are all addressed; the new PRs describe how.

Replaced by:

• core: [core][rest] Support catalog-managed partitions for Format Table reads and writes #8750
• spark: [spark] Support partition DDL for catalog-managed Format Table partitions #8751