Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,8 @@ The tag push also creates the GitHub release, from the version's CHANGELOG secti

8-byte trailer at EOF: `version(u16 LE) | postscriptLen(u16 LE) | magic(VTXF)`. The postscript
(FlatBuffer, immediately before the trailer) points (offset+length) to the Footer (FlatBuffer),
DType (FlatBuffer), and Layout (FlatBuffer) blobs elsewhere in the file.
DType (FlatBuffer), Layout (FlatBuffer) and file statistics (FlatBuffer: one `ArrayStats` per
top-level field, Rust 0.90's per-batch fold, `FileStatsAccumulator`) blobs elsewhere in the file.

Layout tree: `Struct → Zoned(Stats) → Chunked → [Flat, Flat, ...]`
- **Flat** single encoded segment · **Chunked** sequence of Flats · **Struct** one child/column
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- A struct with duplicate field names now fails with `VortexException` when it is built, at any nesting level; the writer used to check only top-level names. ([#696](https://github.com/dfa1/vortex-java/pull/696))
- **Breaking:** a `Double`, `Float` or `BigDecimal` filter operand on an integer column fails with `VortexException`, as vortex-jni does, instead of being truncated so that `x >= 1.5` matched `x = 1`. ([#701](https://github.com/dfa1/vortex-java/issues/701))
- A `vortex.variant` column whose shredded dtype has a control character in a field or variant name fails with `VortexException` instead of `IllegalArgumentException`. ([#694](https://github.com/dfa1/vortex-java/issues/694))
- Written files carry the per-column file statistics (min, max, sum, null and NaN counts) in the footer, as vortex-jni writes them. ([#664](https://github.com/dfa1/vortex-java/issues/664))

### Added
- JDK Flight Recorder events for file open, HTTP range requests, segment decode and encode, chunk filtering, writer back-pressure and the compressor's scheme choices, which mirror Rust's compressor trace. ([#601](https://github.com/dfa1/vortex-java/issues/601))
Expand Down
6 changes: 5 additions & 1 deletion docs/explanation.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,8 +254,11 @@ self-describing tail. A reader bootstraps from the last 8 bytes — no scanning
│ │ leaves point into Footer's
│ │ SegmentSpec[] by index.
├──────────────────────────────────────────────┤
│ Statistics (FlatBuffer) │ ← min/max/sum/null count per
│ │ top-level column.
├──────────────────────────────────────────────┤
│ Postscript (FlatBuffer) │ ← (offset,length) of Footer,
│ │ DType, Layout above.
│ │ DType, Layout, Statistics.
├──────────────────────────────────────────────┤
│ Trailer (8 bytes, little-endian) │
│ u16 version │ u16 postscriptLen │ "VTXF" │ ← magic confirms file type;
Expand Down Expand Up @@ -501,6 +504,7 @@ writer.close()
└─ write DType blob (FlatBuffer)
└─ write Footer blob (FlatBuffer) → SegmentSpec[] + ArraySpec[]
└─ write Layout blob (FlatBuffer) → Struct → Zoned(Stats) → Chunked → [Flat …]
└─ write Statistics (FlatBuffer) → per-column min/max/sum/null count, folded per batch
└─ write Postscript (FlatBuffer) → blob offsets + lengths
└─ write 8-byte trailer → version · postscriptLen · magic (VTXF)
```
Expand Down
4 changes: 3 additions & 1 deletion docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -849,6 +849,8 @@ version (u16 LE) | postscriptLen (u16 LE) | magic ("VTXF")
```

The postscript is a FlatBuffer blob immediately before the trailer. It points (offset + length) to:
the Footer (FlatBuffer), the DType (FlatBuffer), and the Layout (FlatBuffer) — each stored elsewhere in the file.
the Footer (FlatBuffer), the DType (FlatBuffer), the Layout (FlatBuffer) and the file statistics
(FlatBuffer, one `ArrayStats` per top-level column: min, max, sum, null and NaN counts, as vortex-jni
writes them) — each stored elsewhere in the file.

See [explanation.md#memory-model](explanation.md#memory-model) for the mmap lifecycle.

Large diffs are not rendered by default.

Loading
Loading