Repository navigation
[Bug] Deleting a branch's copy of a tag lets its expiration delete main's files #10883
Description
Search before asking
- I searched in the issues and found nothing similar.
Paimon version
2.1.0 and master
Compute Engine
Spark 4.1.4 (spark-sql) on Java 21, with paimon-spark-4.1_2.13 and the paimon catalog on a local warehouse as in the Spark quick start. The Java API does the same through deleteTag and newExpireSnapshots on the branch table.
Minimal reproduce step
Currently, a branch created from a tag starts on a copy of main's snapshot, and expire_snapshots on the branch checks only the branch's own tags, so once the branch's copy of the tag is gone it deletes files main still reads. Keeping that copy leaves main readable.
USE paimon;
CREATE TABLE t (k INT);
INSERT INTO t VALUES (1), (2);
CALL sys.create_tag(table => 't', tag => 't1');
CALL sys.create_branch(table => 't', branch => 'b', tag => 't1');
INSERT INTO `t$branch_b` VALUES (3);
CALL sys.delete_tag(table => '`t$branch_b`', tag => 't1');
CALL sys.expire_snapshots(table => '`t$branch_b`', retain_max => 1, retain_min => 1);
CREATE TABLE u (k INT);
INSERT INTO u VALUES (1), (2);
CALL sys.create_tag(table => 'u', tag => 't1');
CALL sys.create_branch(table => 'u', branch => 'b', tag => 't1');
INSERT INTO `u$branch_b` VALUES (3);
CALL sys.expire_snapshots(table => '`u$branch_b`', retain_max => 1, retain_min => 1);
-- quit and start a new spark-sql session, since this one still reads t from its cache
USE paimon;
SELECT * FROM t;
-- java.io.FileNotFoundException: File '.../t/manifest/manifest-list-...-0' not found, should be 1 and 2
SELECT * FROM t VERSION AS OF 't1';
-- the same FileNotFoundException, though main's tag t1 is still there
SELECT * FROM u;
-- 1 and 2, so with b's copy of t1 kept the same expiration leaves main aloneWhat doesn't meet your expectations?
Both reads of t return 1 and 2, since main's snapshot 1 never expired and main's tag t1 still exists.
Anything else?
replace_tag on b, or b's copy expiring through time_retained, does the same. On a primary-key table, a compact on b first also deletes a data file main's newer snapshots read.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity