Skip to content

[python] Check option changes on tables with snapshots like Java - #10856

Merged
JingsongLi merged 1 commit into
apache:masterfrom
plusplusjiajia:fix/python-alter-option-guards
Oct 29, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
plusplusjiajia:fix/python-alter-option-guards

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Purpose

On a table with snapshots, PyPaimon only rejected changes to the immutable options. Java SchemaManagerUtils.checkAlterTableOption and checkResetTableOption also reject:

  • changing bucket from or to -1, and resetting bucket;
  • changing deletion-vectors.enabled unless deletion-vectors.modifiable is true;
  • changing ignore-delete, ignore-update-before or data-evolution.nested-field.enabled from true to false;
  • changing or resetting clustering.columns when pk-clustering-override is set.

Accepting them re-buckets or merges existing keys differently, which can produce duplicate rows in primary-key tables.

These checks are now ported with Java's messages, and bucket-function.provider is added to the immutable options as in Java. As in Java:

  • the checks compare with the table's options before the batch, while changes still apply in order within the batch;
  • the rejections, including the existing immutable option rejection, raise NotImplementedError, Python's counterpart of UnsupportedOperationException, instead of ValueError;
  • bucket is parsed like Integer.parseInt.

Option values are compared as PyPaimon reads them, including native values, enums and boolean spellings such as yes.

Tests

  • TestAlterTableOptions in schema_manager_test.py: immutable options, bucket changes and parsing, deletion vectors, options that cannot be disabled, clustering columns, the checks only applying with snapshots, native option values, comparison with the original options within a batch, and unchanged values written differently.
  • FileSystemCatalogTest#test_alter_immutable_options: the catalog rejects these changes without creating a new schema.
  • The 11 tests fail on master. The schema manager and catalog tests pass locally: 80 tests. flake8 is clean.

On a table with snapshots, PyPaimon only rejected changes to the
immutable options. Java SchemaManagerUtils.checkAlterTableOption and
checkResetTableOption also reject, for example, changing bucket from or
to -1 and resetting bucket, changing deletion-vectors.enabled without
deletion-vectors.modifiable, and changing ignore-delete or
ignore-update-before from true to false. Accepting them re-buckets or
merges existing keys differently, which can produce duplicate rows in
primary-key tables.

These checks are now ported with Java's messages, and
bucket-function.provider is added to the immutable options as in Java.
As in Java, the checks compare with the table's options before the
batch, and the rejections raise NotImplementedError, Python's
counterpart of UnsupportedOperationException; changes still apply in
order within a batch. Option values are compared as PyPaimon reads them,
including native values, enums and boolean spellings such as yes.
@plusplusjiajia
plusplusjiajia marked this pull request as ready for review October 28, 2026 16:22
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 9895baa into apache:master Oct 29, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants