[Bug] Spark partition metadata queries fail for chain tables and query-authorized reads #10837

Search before asking

• I searched in the issues and found nothing similar.

Related fixes are listed below; they do not cover partition metadata on chain or query-authorized splits.

Paimon version

2.0 & master 2.2-SNAPSHOT.

This revision includes #10725, which preserves query authorization requirements through fallback and chain scans and separates internal partition selection from user partition filters.

Compute Engine

Apache Spark 3.5.8 and JDK 11.

Minimal reproduce step

The chain-table case does not require query authorization, column masking, or a REST catalog. The SQL below is reduced from the setup in the existing SparkChainTableMetadataTest, which currently asserts that chain partition metadata queries fail.

Start a Spark session with the Paimon Spark connector, its session extensions, and a filesystem catalog named spark_test. For example, use these session startup options with a fresh warehouse directory:

--conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
--conf spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkCatalog
--conf spark.sql.catalog.spark_test.warehouse=file:///tmp/paimon-issue-c-warehouse

Create a chain table with a snapshot and a later delta:

CREATE DATABASE spark_test.issue_c;

CREATE TABLE spark_test.issue_c.orders (
    id INT,
    payload STRING,
    seq BIGINT,
    dt STRING
)
USING paimon
PARTITIONED BY (dt)
TBLPROPERTIES (
    'primary-key' = 'dt,id',
    'bucket-key' = 'id',
    'bucket' = '1',
    'sequence.field' = 'seq',
    'chain-table.enabled' = 'true',
    'partition.timestamp-pattern' = '$dt',
    'partition.timestamp-formatter' = 'yyyyMMdd'
);

CALL spark_test.sys.create_branch('issue_c.orders', 'snapshot');
CALL spark_test.sys.create_branch('issue_c.orders', 'delta');

ALTER TABLE spark_test.issue_c.orders SET TBLPROPERTIES (
    'scan.fallback-snapshot-branch' = 'snapshot',
    'scan.fallback-delta-branch' = 'delta'
);
ALTER TABLE spark_test.issue_c.`orders$branch_snapshot` SET TBLPROPERTIES (
    'scan.fallback-snapshot-branch' = 'snapshot',
    'scan.fallback-delta-branch' = 'delta'
);
ALTER TABLE spark_test.issue_c.`orders$branch_delta` SET TBLPROPERTIES (
    'scan.fallback-snapshot-branch' = 'snapshot',
    'scan.fallback-delta-branch' = 'delta'
);

INSERT INTO spark_test.issue_c.`orders$branch_snapshot`
VALUES (1, 'old', 1, '20250810'), (2, 'keep', 1, '20250810');

INSERT INTO spark_test.issue_c.`orders$branch_delta`
VALUES (1, 'new', 2, '20250811');

-- Ordinary columns: read the merged logical partition.
SELECT id, payload, dt
FROM spark_test.issue_c.orders
WHERE dt = '20250811'
ORDER BY id;

-- Partition metadata on the snapshot-only chain read fails.
SELECT id, __paimon_partition
FROM spark_test.issue_c.orders
WHERE dt = '20250810';

-- Partition metadata on the snapshot + delta chain read also fails.
SELECT id, dt, __paimon_partition.dt AS metadata_dt
FROM spark_test.issue_c.orders
WHERE dt = '20250811';

-- A metadata-only projection and a metadata predicate encounter the same restriction.
SELECT __paimon_partition.dt
FROM spark_test.issue_c.orders
WHERE dt = '20250811';

SELECT id
FROM spark_test.issue_c.orders
WHERE dt = '20250811'
  AND __paimon_partition.dt = '20250811';

The failure is:

java.lang.UnsupportedOperationException:
Partition metadata is only supported for DataSplit without query authorization.

There is a second affected path: an ordinary table read with authorization rules. Given a REST catalog that implements query authorization, enable query-auth.enabled on an ordinary partitioned table and grant the querying user access to id and dt. Return a row-filter policy such as id > 0, without column masks, and ensure the table contains a matching row. Reading id, dt is permitted, but adding __paimon_partition.dt encounters the same restriction because the scan produces QueryAuthSplit(DataSplit). This case does not require a chain table or a masking policy.

For ordinary tables, merely enabling authorization without any row-filter or masking rules does not trigger this split-type failure: the scan retains its original DataSplit. A valid authorization service and a policy that produces the wrapper are required for this second case.

What doesn't meet your expectations?

Queries should be able to project and filter __paimon_partition on chain tables and query-authorized reads, with the following behavior:

• For a chain read, metadata should describe the logical partition of the returned row. In the merged 20250811 partition above, both the updated row and the row retained from the older snapshot should have dt = metadata_dt = '20250811'.
• With query authorization, metadata must expose the same visible partition values as the corresponding ordinary columns. For example, if dt is masked to '***', both dt and __paimon_partition.dt should return '***'. This also applies to masks that depend on another column or return NULL.
• Accessing __paimon_partition.dt, including only in a predicate, must require permission on dt and must not bypass a denied column through metadata. With Spark's default spark.sql.optimizer.nestedSchemaPruning.enabled=true, accessing a single metadata subfield must not require permission on unrelated partition fields. Reading the complete partition struct must authorize all its exposed fields.
• Row filters must still apply. Predicates on partition metadata, and queries combining them with COUNT(*) or LIMIT, should operate on the visible values without exposing or filtering against raw masked partition values.

Scope when nested schema pruning is disabled: with spark.sql.optimizer.nestedSchemaPruning.enabled=false, Spark requests the complete partition struct even when the query only accesses .dt. Authorization therefore requires all partition fields, including fields such as region that the query does not explicitly access. Reads, row filters and column masks remain supported when those permissions are available. Precise subfield authorization with this setting disabled is a separate follow-up and is outside the scope of this issue's proposed fix.

For example, after configuring a mask of dt -> '***', this query should compare the metadata with the masked value:

SELECT id, dt, __paimon_partition.dt AS metadata_dt
FROM spark_test.issue_c.orders
WHERE __paimon_partition.dt = '***';

This does not require changing the existing restriction on explicit physical partition filters over masked columns.

Anything else?

The reader currently rejects partition metadata for chain and query-authorized splits. Metadata must come from visible rows to preserve logical partition values and column masks.

Local validation of the proposed fix: 22 cases passed on each of Spark 3.2.4 and 3.3.4 (JDK 11), and Spark 4.0.3 and 4.1.2 (JDK 17), using real Spark reads with test-catalog authorization policies.

Related PRs, which do not cover this partition metadata support:

• #10481: partition and bucket metadata for ordinary merge reads.
• #10652: chain bucket metadata; retains the partition metadata restriction.
• #10725: core authorization and internal partition-selection prerequisites.

Are you willing to submit a PR?

• I'm willing to submit a PR!