You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
DictLayoutDecoder: lazy dict rejects U8/U16/I8/I16 value ptypes (hit by real Python-written files) #206
Found by the Raincloud conformance PoC (#205). Files written by the Vortex Python bindings (vortex-data==0.69.0) dict-encode narrow-integer columns; the dict values land as U8/U16 primitives. DictLayoutDecoder.buildLazyDictPrimitive only handles I64/U64/I32/U32/F64/F32 and throws for the rest:
VortexException: vortex.dict: layout: unsupported ptype for lazy dict: U8
at DictLayoutDecoder.buildLazyDictPrimitive(DictLayoutDecoder.java:137)
Hit by 2 of 4 PoC datasets (kepler-exoplanet-search-results U8, palmer-archipelago-antarctica-penguin-data U16), so narrow dict values are common in real-world data, not a corner case.
Repro: build either slug with Raincloud (python -m scripts.pipeline.build <slug>), then vortex-cli export <slug>.vortex.
Fix direction: extend the switch (and the lazy DictXxxArray family) to byte/short value ptypes, mirroring what the eager path supports.
Fix implemented: DictByteArray / DictShortArray mirror the existing lazy dict records (raw-signed storage; unsigned widening stays in ByteArray.getInt/ShortArray.getInt via dtype), and DictLayoutDecoder.buildLazyDictPrimitive gained the I8, U8 / I16, U16 arms. Unit tests cover all four code widths per record plus the U8>127 / U16>32767 unsigned corners; full ./mvnw verify green. On the Raincloud corpus this flips uci-bike-sharing-dataset to a full value-for-value pass and unblocks kepler/penguins scans far enough to expose #210 and #211 (reclassified in the matrix). Closing when the commit lands on main.
Activity
dfa1 commentedon Jul 6, 2026
Fix implemented:
DictByteArray/DictShortArraymirror the existing lazy dict records (raw-signed storage; unsigned widening stays inByteArray.getInt/ShortArray.getIntvia dtype), andDictLayoutDecoder.buildLazyDictPrimitivegained theI8, U8/I16, U16arms. Unit tests cover all four code widths per record plus the U8>127 / U16>32767 unsigned corners; full./mvnw verifygreen. On the Raincloud corpus this flipsuci-bike-sharing-datasetto a full value-for-value pass and unblocks kepler/penguins scans far enough to expose #210 and #211 (reclassified in the matrix). Closing when the commit lands on main.test: add Raincloud real-world conformance harness (#205)
fix: lazy dict decode for I8/U8/I16/U16 value columns (#206)
test: add Raincloud real-world conformance harness (#205)
fix: lazy dict decode for I8/U8/I16/U16 value columns (#206)
test: add Raincloud real-world conformance harness (#205)
fix: lazy dict decode for I8/U8/I16/U16 value columns (#206)