Skip to content

DictLayoutDecoder: lazy dict rejects U8/U16/I8/I16 value ptypes (hit by real Python-written files) #206

Closed

Description

@dfa1

Found by the Raincloud conformance PoC (#205). Files written by the Vortex Python bindings (vortex-data==0.69.0) dict-encode narrow-integer columns; the dict values land as U8/U16 primitives. DictLayoutDecoder.buildLazyDictPrimitive only handles I64/U64/I32/U32/F64/F32 and throws for the rest:

VortexException: vortex.dict: layout: unsupported ptype for lazy dict: U8
    at DictLayoutDecoder.buildLazyDictPrimitive(DictLayoutDecoder.java:137)

Hit by 2 of 4 PoC datasets (kepler-exoplanet-search-results U8, palmer-archipelago-antarctica-penguin-data U16), so narrow dict values are common in real-world data, not a corner case.

Repro: build either slug with Raincloud (python -m scripts.pipeline.build <slug>), then vortex-cli export <slug>.vortex.

Fix direction: extend the switch (and the lazy DictXxxArray family) to byte/short value ptypes, mirroring what the eager path supports.

Activity

  1. added
    bugSomething isn't working
    on Jul 6, 2026
  2. dfa1 commented on Jul 6, 2026

    @dfa1
    OwnerAuthor

    Fix implemented: DictByteArray / DictShortArray mirror the existing lazy dict records (raw-signed storage; unsigned widening stays in ByteArray.getInt/ShortArray.getInt via dtype), and DictLayoutDecoder.buildLazyDictPrimitive gained the I8, U8 / I16, U16 arms. Unit tests cover all four code widths per record plus the U8>127 / U16>32767 unsigned corners; full ./mvnw verify green. On the Raincloud corpus this flips uci-bike-sharing-dataset to a full value-for-value pass and unblocks kepler/penguins scans far enough to expose #210 and #211 (reclassified in the matrix). Closing when the commit lands on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions