You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
InspectorTree.build()'s usedEncodings set (surfaced by the CLI's inspect command) only reports the rootArrayNode's encoding for each Flat layout segment — it does not recursively walk into encodings nested inside that same segment's buffer. This makes encodings like vortex.fsst invisible in the inspector even when they're genuinely selected and used by the writer.
Repro
Import any dataset with a high-cardinality Utf8 column through ParquetImporter/VortexWriter with WriteOptions.cascading(3) (e.g. NYC 311 complaints' "Incident Address" column). The cost-based dispatch in CascadingCompressor correctly picks vortex.fsst for such columns (confirmed by instrumenting MaskedEncodingEncoder.encodeValues directly — the real write path logs picked=vortex.fsst, and the on-disk size matches FSST's expected output almost exactly).
No vortex.fsst anywhere, and the Layout: tree's bracketed encoding list for that column also omits it — even though the values genuinely are FSST-compressed.
Root cause
inspector/src/main/java/io/github/dfa1/vortex/inspect/InspectorTree.java's peekFlatRoot(MemorySegment seg, List<EncodingId> arraySpecs) reads only the rootFbsArrayNode of a Flat segment's trailing FlatBuffer:
It never descends into root.children(). This is fine for encodings that are themselves separate layout nodes with their own segments (e.g. vortex.dict, whose dictionary-values child is a distinct Flat layout node, independently peekable) — but any encoding that lives purely inside one Flat segment's ArrayNode tree (FSST, ALP, bitpacked when not chunked separately, etc.) is invisible to this shallow peek.
Impact
Purely a diagnostic/observability gap — VortexReader.scan() decodes the full ArrayNode tree correctly regardless, so this does not affect correctness of reads.
But it makes the CLI inspect command (and anything built on InspectorTree.usedEncodings(), e.g. the JavaVsJniFsstBenchmark's verifyFsstSelected check) unreliable for auditing which encoding a real file's column actually uses whenever that encoding is nested rather than a top-level layout wrapper.
Expected
InspectorTree should recursively walk a Flat segment's full ArrayNode tree (not just its root) when computing usedEncodings/the per-node encoding list, so encodings like vortex.fsst are reported accurately. This likely needs peekFlatRoot (or a new sibling helper) to walk root.children() using the same arraySpecs resolution, accumulating every encoding id encountered rather than stopping at the root.
Where
inspector/src/main/java/io/github/dfa1/vortex/inspect/InspectorTree.java — peekFlatRoot, and whichever caller(s) build the per-NodeusedEncodings set from that peek result.
Activity
fix(inspector): walk full ArrayNode tree for usedEncodings (#298)