Skip to content

Releases

version 1.9.1

Choose a tag to compare

@rhpvorderman rhpvorderman released this 04 Nov 12:53
2498167
  • Fix BrokenPipeError when piping python -m isal.igzip into head,
    less or similar programs.

version 1.9.0

Choose a tag to compare

@rhpvorderman rhpvorderman released this 03 Nov 11:57
6202da6

By far most of the work on this release has been done by
@ptweezy. Almost all bug fixes and improvements
are their work.

Major changes:

  • Updated included ISA-L library to 2.32.1.
  • Python 3.10 and PyPy 3.10 are no longer supported. Added support for PyPy
    3.12 and Python 3.15.
  • Restore PyPy wheel builds.
  • Fix builds on macOS with Apple Silicon, which used the portable C code of
    ISA-L rather than its optimized aarch64 assembly. On an Apple M4,
    decompression is 1.6 to 3 times faster, CRC-32 is 50 times faster, and
    Adler-32 is 7 times faster.

Optimizations and features:

  • isal_zlib.compressobj supports strategy=Z_FIXED at compression
    level 0. Level 0 then uses fixed Huffman codes instead of starting each
    block with a Huffman table of about 110 bytes. This shrinks small flushed
    writes, such as websocket messages: JSON messages of about 440 bytes, each
    followed by flush(Z_SYNC_FLUSH), compress to an average of 75 instead
    of 188 bytes. Blocks of a few KiB or more can be larger: about 5% for JSON
    and 10% for Python source code at 64 KiB. Other strategies, and Z_FIXED
    at levels 1 to 3, still give a warning and use the default strategy.
  • Speed up import isal.igzip and import isal.igzip_threaded by
    importing argparse and shutil only when the command line interface
    runs, and by dropping the multiprocessing import.
  • Speed up reading lines from binary igzip files through iteration,
    readlines(), and readline(). On an Apple M4, reading FASTQ data is
    1.13 times faster, and reading lines of a few bytes is 1.5 times faster.
  • Speed up each write() call to binary igzip files on CPython. On an
    AMD Ryzen 7 9800X3D with Python 3.14.8, writing FASTQ data or Python source
    code one line at a time is 1.1 times faster.
  • Speed up many small writes to igzip.IGzipFile on Python 3.12 and later.
    IGzipFile collects these writes in the write buffer of
    gzip.GzipFile and compresses them in large chunks, which makes writing
    lines 1.4 times faster.

Bug fixes:

  • Fix zlib streams compressed with a preset dictionary so they can be decoded
    by standard zlib, including builds linked against older ISA-L releases.
  • Fix backward seeking in igzip files by discarding buffered compressed data
    when rewinding the underlying file.
  • Fix a crash when igzip.decompress gets an object that is not
    bytes-like, such as a str. It raises a TypeError instead.
  • Fix a crash after creating igzip_lib.IgzipDecompressor objects with a
    zdict.
  • Fix error handling when more than one interpreter imports
    isal.isal_zlib or isal.igzip_lib, for example, the main interpreter
    and a legacy subinterpreter that Py_NewInterpreter() creates. Each
    import replaced the error classes that the C code raises, so
    except isal_zlib.error, except igzip_lib.error, and
    except gzip.BadGzipFile caught errors only in the interpreter that
    imported the modules last.
  • Fix a hang in isal_zlib.Compress: after flush() raised an error for
    an unsupported mode, the next call on the same object never returned.
  • Fix a crash when you call flush(Z_FINISH) twice on a level 0
    isal_zlib.Compress object.
  • Fix unused_data of isal_zlib.Decompress and
    igzip_lib.IgzipDecompressor, which could contain bytes from outside the
    input buffer when max_length split the end of a stream. At the end of a
    stream, isal_zlib.Decompress.unconsumed_tail is empty, matching the
    CPython fix for gh-158169.
  • Fix igzip_lib.IgzipDecompressor.needs_input, which could be True
    after decompress() returned max_length bytes while more output was
    still available. Readers that stop at the end of the file when
    needs_input is True, such as CPython's DecompressReader, raised
    EOFError for valid data.
  • Fix isal_zlib.decompressobj with a wbits of 40 to 47, which detects
    zlib or gzip data. It failed on gzip data when the first decompress()
    call got fewer than 2 bytes, for example, when the data arrived byte by
    byte.
  • Fix isal_zlib.decompressobj with a wbits of 40 to 47 after a
    flush() call that came before any data. It then failed on both zlib and
    gzip data.
  • igzip_threaded.open raises ValueError for a block_size of about
    3.6 GiB or more and TypeError for a compresslevel that is not an
    integer. Both raised SystemError before.
  • Fix igzip_threaded writers at compression level 0, which raised
    OverflowError for data that does not compress, such as random bytes or
    already compressed files. At compression level 0, block_size must now be
    less than about 2 GiB.
  • Extraordinarily large gzip headers with either a very long comment or
    filename will now cause an OverflowError rather than use more and more
    memory. The size is limited to the internal buffer size which is currently
    set at 512 KiB.
  • Fix igzip_threaded readers and writers that waited for a polling timeout
    of up to 50 ms at the end of the stream or when closed. Writing and closing
    a small file takes less than a millisecond instead of 60 ms.
  • Fix igzip_threaded writers that waited forever when writing to the
    underlying file failed, for example, because the disk was full. A later call
    to write(), flush(), or close() raises the error. After an error,
    close() no longer closes a file object that you passed to
    igzip_threaded.open.
  • Fix igzip_threaded writers with more than one thread that waited
    forever when compressing a block failed, for example, because there was not
    enough memory. A later call to write(), flush(), or close()
    raises the error.
  • Fix a crash on the free-threaded build when one thread reads
    unused_data or unconsumed_tail of an isal_zlib decompressor
    while another thread decompresses with it. Reading the attributes of
    isal_zlib decompressors and igzip_lib.IgzipDecompressor from other
    threads is now safe, and so is calling decompress() from several
    threads at the same time when wbits detects the format.
  • isal_zlib.crc32_combine and the read() and seek() methods of the
    private isal_zlib._GzipReader class raise TypeError instead of
    SystemError for arguments of the wrong type or number.
    igzip.IGzipFile was not affected.
  • Fix a crash on PyPy when you create an isal_zlib.Compress or
    isal_zlib.Decompress object by calling the type, for example,
    isal_zlib.Compress(), and then use it. Calling the type raises
    TypeError now, as it does on CPython. Use compressobj() and
    decompressobj() instead.
  • Fix readinto() of the private isal_zlib._GzipReader class, which
    wrote into read-only objects such as bytes. It raises TypeError now.
    igzip.IGzipFile was not affected.
  • Fix the mtime attribute of igzip.IGzipFile objects in read mode,
    which was None for gzip files with a modification time of 0. It is 0 now,
    as in the gzip module. gzip -n, python -m isal.igzip -n, and
    igzip_threaded writers make such files, and so does gzip.compress by
    default on Python 3.14.
  • Fix a crash when there is not enough memory to create an
    igzip_lib.IgzipDecompressor. It raises MemoryError now.
  • Fix a crash when there is not enough memory to raise BadGzipFile for
    data that is not in the gzip format, for example, in igzip.decompress.
    It raises MemoryError now.
  • Fix a crash in igzip_lib.IgzipDecompressor.decompress() after a call on
    the same object raised MemoryError. The next call used the input of the
    failed call after the caller had released it. This matches the CPython fix
    for gh-148395.
  • Fix a crash on macOS and PyPy when there is not enough memory to create the
    internal lock of an isal_zlib.Compress, isal_zlib.Decompress, or
    igzip_lib.IgzipDecompressor object. It raises MemoryError now.
  • Fix the -0 and --fast options of python -m isal.igzip, which
    compressed at level 2 instead of level 0.
  • Fix crashes and wrong data when one thread calls seek() of the
    private isal_zlib._GzipReader class while another thread reads from the
    same object. igzip.IGzipFile was not affected.

version 1.8.0

Choose a tag to compare

@rhpvorderman rhpvorderman released this 10 Sep 08:51
  • Python 3.14 is supported.
  • Python 3.8 and 3.9 are no longer supported.
  • Fix an issue where flushing using igzip_threaded caused a gzip end of stream
    and started a new gzip stream. In essence creating a concatenated gzip
    stream. Now it is in concordance with how single threaded gzip streams
    are flushed using Z_SYNC_FLUSH.
  • Change build backend to setuptools-scm which is more commonly used and
    supported.
  • Include test packages in the source distribution, so source distribution
    installations can be verified.
  • Fix an issue where some tests failed because they ignored PYTHONPATH.
  • Enable support for free-threading and build free-threaded wheels for
    CPython 3.14. Thanks to @lysnikolaou and @ngoldbaum.

version 1.7.2

Choose a tag to compare

@rhpvorderman rhpvorderman released this 05 Mar 12:18
  • Use upstream ISA-L version 2.31.1 which includes patches to make
    installation on MacOS ARM64 possible.
  • Fix a bug where bytes were copied in the wrong order on big endian
    architectures. Fixes test failures on s390x.
  • Enable building on GNU/Hurd platforms.

version 1.7.1

Choose a tag to compare

@rhpvorderman rhpvorderman released this 25 Sep 13:37
  • Fix a bug where flushing files when writing in threaded mode did not work
    properly.
  • Prevent threaded opening from blocking python exit when an error is thrown
    in the calling thread.

version 1.7.0

Choose a tag to compare

@rhpvorderman rhpvorderman released this 09 Aug 14:22
  • Include a patched ISA-L version 2.31. The applied patches make compilation
    and wheelbuilding on MacOS ARM64 possible.
  • Fix a bug where READ and WRITE in isal.igzip were inconsistent with the
    values in gzip on Python 3.13
  • Small simplifications to the igzip.compress function, which should lead
    to less overhead.

version 1.6.1

Choose a tag to compare

@rhpvorderman rhpvorderman released this 11 Mar 13:09
aa4a483
  • Fix a bug where streams that were passed to igzip_threaded.open where closed.

Version 1.6.0

Choose a tag to compare

@rhpvorderman rhpvorderman released this 19 Feb 09:02
ffe04cb
  • Fix a bug where compression levels for IGzipFile where checked in read mode.
  • Update statically linked ISA-L release to 2.31.0
  • Fix an error that occurred in the __close__ function when a threaded
    writer was initialized with incorrect parameters.

Version 1.5.3

Choose a tag to compare

@rhpvorderman rhpvorderman released this 24 Nov 08:36
f380197
  • Fix a bug where append mode would not work when using
    igzip_threaded.open.

Version 1.5.2

Choose a tag to compare

@rhpvorderman rhpvorderman released this 03 Nov 15:44
b4d1be1
  • Fix a bug where a filehandle remained opened when igzip_threaded.open
    was used for writing with a wrong compression level.
  • Fix a memory leak that occurred when an error was thrown for a gzip header
    with the wrong magic numbers.
  • Fix a memory leak that occurred when isal_zlib.decompressobj was given a
    wrong wbits value.