[plusplusjiajia]

Purpose

PyArrow implements PythonFile.read_at(nbytes, offset) as seek(offset) plus read(nbytes) on the wrapped file object, so JindoInputFile.read_at, which calls Jindo's pread, is never reached. Every Parquet column chunk and manifest range read goes through Jindo's sequential stream, which reads ahead a whole buffer. Against OSS, 40 positional reads of 12 KB transferred 58 MB, and reading 3 of 22 columns transferred 33% more than PyArrow S3.

This PR adds fs.oss.jindo.pread-max-size, disabled by default. When it is set, JindoInputFile tracks its own position. A read of at most that size uses pread if it follows a backward seek or a forward seek of more than 1 MiB, or if it is the first read on the file and not at the stream position.

All other reads keep the stream. Contiguous reads and short forward skips are served by Jindo's read-ahead, and reads above the threshold keep Jindo's parallel download. With the default, read, seek and tell take the existing code path.

As with other options, a value in the REST data token overrides the catalog option, so a REST server can enable it per table.

Measured on Alibaba Cloud OSS with pyjindosdk 6.10.4. Bytes come from the OSS access logs; times include catalog and planning work and vary between runs:

Read	PyArrow S3	Jindo, master	Jindo, this PR, default	Jindo, this PR, 4mb
40 positional reads of 12 KB, 2 MB apart	0.49 MB, 2.39 s	58.46 MB, 4.01 s	58.46 MB, 3.45 s	0.49 MB, 2.26 s
3 of 22 columns, 16 key lookups	38.11 MB, 8.16 s	50.56 MB, 10.30 s	50.56 MB, 9.70 s	38.11 MB, 8.70 s
22 of 22 columns, 16 key lookups	376.36 MB, 11.86 s	380.34 MB in 106 GETs, 11.07 s	380.34 MB in 106 GETs, 10.82 s	376.45 MB in 106 GETs, 11.46 s
100 reads of 4 KB, 8 KB apart	100 GETs, 4.30 s	2 GETs, 1.08 s	2 GETs, 0.99 s	3 GETs, 1.05 s
100 reads of 4 KB, 64 KB apart	100 GETs, 4.24 s	5 GETs, 1.30 s	5 GETs, 1.29 s	6 GETs, 1.31 s

Tests

• Existing tests are unchanged and still pass.
• New tests in pypaimon/tests/jindo_file_system_test.py:
  • the default keeps the original stream behavior;
  • with the option set:
    • a read after a seek fetches exactly its range with pread;
    • a contiguous read after it returns to the stream;
    • sequential reads and short forward skips keep the stream;
    • long forward and backward jumps use pread;
    • reads are clamped to the known file size;
    • SEEK_CUR works;
    • reads above the threshold keep the stream;
    • PyArrow PythonFile.read_at reaches pread;
  • the option is not forwarded to Jindo;
  • the handler passes the option to input files;
  • a value in the REST data token overrides the catalog option.
• jindo_file_system_test.py, video_format_test.py, caching_file_io_test.py and rest/rest_token_file_io_test.py pass (141 passed).
• The table above was measured on OSS. An earlier run with the option at 4 MiB transferred the same bytes in every case.

[JingsongLi]

+1