[spark] Support managed Format Table partition DDL

[sundapeng]

Relationship

Second part of splitting #8713; stacked on the core part #8728 (the diff shown here includes
the core commit until that PR merges — please review only the [spark] Support managed Format Table partition DDL commit). Marked draft until the core part lands.

Purpose

Spark SQL partition DDL for managed Format Tables on top of the core part:
ALTER TABLE ... ADD/DROP PARTITION backed by the catalog.

Main changes

• A partition gateway (FormatTablePartitionCatalog) wired into PaimonFormatTable;
  DDL executes against the catalog, not the filesystem.
• ADD registers then creates the partition directory (empty partitions are immediately
  visible and writable). Partition values are validated against path escapes
  (./.., resolved path must stay under the table root); ADD ... LOCATION is
  rejected (managed table locations are derived).
• DROP unregisters first, then deletes the directory (the service never deletes data);
  a full spec that is not registered fails without touching its directory (protects
  data awaiting registration). Partial specs expand through the catalog to leaf
  partitions; expansions and unregistrations run in bounded batches.
• Strict-ADD atomicity (reject the whole batch if any partition exists) is enforced by
  the catalog service — the client forwards IF NOT EXISTS unchanged and skips Spark's
  client-side existence precheck.
• Session-level metastore.partitioned-table overrides are ignored with a WARN (the
  persisted table property wins) instead of failing table loads.

Tests

DDL planning with a fault-injection catalog (26), partition management round-trips (20),
gateway contract tests incl. mutation batching (6), option handling (5). Partial-spec DROP expansion lists partitions unfiltered and matches client-side — no server-side prefix pushdown in this first version.

[sundapeng]

Closing in favour of a re-split of this work.

The implementation changed enough since this split that updating the branch in
place would replace every line and leave the review comments pointing at code
that no longer exists:

• the serializable provider with its 30s listing cache and JVM-wide generation
  counter is gone, replaced by a partition manager that creates one catalog per
  operation and closes it;
• the managed scan subclass is gone, folded into FormatTableScan;
• the two procedures are gone; MSCK REPAIR TABLE is the only entry point;
• the feature no longer reuses metastore.partitioned-table. It has its own
  option, format-table.partition-source = filesystem | rest, so a
  catalog-level default set for Paimon tables cannot turn it on for a Format
  Table by accident.

The review comments here are all addressed; the new PRs describe how.

Replaced by:

• core: [core][rest] Support catalog-managed partitions for Format Table reads and writes #8750
• spark: [spark] Support partition DDL for catalog-managed Format Table partitions #8751