You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nyc-311: coordinate F64 columns ~34MB larger than vortex-jni (ALP-RD not selected) #304
Follow-up to #299 / PR #302. On nyc-311, vortex-java is now 30 MB smaller than vortex-jni overall, but the single biggest column where java is still larger is a high-precision float coordinate — jni picks ALP-RD, java falls back to dict + alp.
Evidence — per-column bytes (java re-encode vs vortex-jni, nyc-311)
Column
jni MB
java MB
delta
jni encoding
java encoding
Latitude
110.3
139.9
+29.6
vortex.alprd
vortex.dict, vortex.alp
X Coordinate (State Plane)
42.0
46.3
+4.3
fastlanes.for
fastlanes.bitpacked
Longitude
105.1
105.4
+0.3
alp, alprd
alp
~34 MB combined (Latitude dominates). Latitude/Longitude are full-precision F64 coordinates (~40.5–40.9, many mantissa bits). jni encodes Latitude with ALP-RD (ALP real-double: splits the double into a dictionary-coded left part + a bitpacked right part — the scheme ALP falls back to when values don't fit plain ALP's integer form). java instead global-dicts it and ALP-encodes the dict values, which is larger.
Hypothesis
java has an AlpRdEncodingEncoder (used correctly for Longitude, which shows alprd), so the algorithm works — this is an encoder-selection / cost gap for Latitude specifically:
Latitude qualifies as a global-dict candidate (enough repeated rounded coordinates to pass the ratio gate) and the dict path is chosen before ALP-RD gets a fair cost comparison; or
the ALP-RD sample-cost estimate loses to dict+ALP on the sample even though it wins on the full column.
Worth checking why Latitude and Longitude — near-identical columns — diverge (dict+alp vs alprd).
VortexWriter.isDictCandidate (F64 admission) + the cascade competition order (Dict vs AlpRd for F64)
Context
Measured on the hydrated Raincloud nyc-311 corpus. Fuzzier than #303 (a selection-quality question, not a raw-child-encoding gap). Not started — analysis only.
Activity
feat(writer): add ALP-RD to the cascade competition (#304)
fix(writer): train ALP-RD dictionary on a stratified sample