Repository navigation
Deprecate and schedule removal of collections.abc.ByteString and typing.ByteString #91896
Description
Activity
The general rule is that we can never add methods to ABCs once they are published. The purpose of an ABC is promise that a minimal set of methods are available. If
isinstance(x, SomeABC)returns true, the methods in the ABC are expected to be present. Adding methods presents a problem for existing code that has registered a class as being compliant with SomeABC. If it lacks the new methods, then the registered promise is invalid.Perhaps because this is an empty ABC that is almost entirely unused, this might be okay. On the other hand, because it is an empty ABC that is almost entirely unused, there is almost nothing to be gained by adding it. AFAICT no one has ever asked for or needed this since the ABC was added 15 years ago. That suggests that there is no problem to be solved here.
Perhaps because this is an empty ABC that is almost entirely unused, this might be okay.
Right, the ABC is hardly useful with no methods. I searched on
grep.appand found no uses of ByteString.register except for one ofmemoryviewin what appears to be an old copy oftyping.py.On the other hand, because it is an empty ABC that is almost entirely unused, there is almost nothing to be gained by adding it. AFAICT no one has ever asked for or needed this since the ABC was added 15 years ago. That suggests that there is no problem to be solved here.
Well, I'm asking for it now. The use case is type annotating code that accepts both
bytesandbytearray.If you were to do it correctly, the procedure would be to subclass ByteString and add the new methods in a subclass. Loosely, this is similar to why we had to add IterableUserDict in Python 2 rather than modifying the existing UserDict.
To annotate code that accepts both bytes or bytearray wouldn't the correct way be to write:
b: bytes | bytearrayjust like you would withtuple | list?One other thought: the name ByteString wasn't very good to begin as is suggests
bytes | str. If you really must do this, it would be better to create a new, well-named ABC with the requisite methods and even leave the old ABC around (it isn't hurting anything) or deprecate it.I forgot to mention that the docs promise something narrower that the bytes/bytearray API. It says, "ABCs for read-only and mutable sequences". There is no promise of the extra methods found in bytes or bytearray. Existing code reasonably by subclassing from ByteString and not providing or expecting any of the stringlike methods.
All around, I think this proposal is a contract violation and that a new ABC should be created. A scan on grep.app is insufficient to show this won't be a breaking change (most of the world's Python code isn't publicly visible).
I forgot to mention that the docs promise something narrower that the bytes/bytearray API. It says, "ABCs for read-only and mutable sequences". There is no promise of the extra methods found in bytes or bytearray. Existing code reasonably by subclassing from ByteString and not providing or expecting any of the stringlike methods.
That documentation is for Sequence, MutableSequence, and ByteString together (https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence). "Read-only and mutable sequences" is a good description for the first two, but the documentation really doesn't tell me what ByteString is good for.
ByteString is also documented at https://docs.python.org/3/library/typing.html#typing.ByteString, but that documentation has a couple of problems:
memoryviewis not in fact registered as a ByteStringByteStringis not in fact generic (and it doesn't make sense for it to be generic)
All around, I think this proposal is a contract violation and that a new ABC should be created. A scan on grep.app is insufficient to show this won't be a breaking change (most of the world's Python code isn't publicly visible).
That's a reasonable point. If we can't use the existing ByteString ABC for
bytes | bytearray, I don't think it's worth creating a new ABC—as you said above, the union annotation is good enough.But if we keep ByteString as is, with no methods, I have no idea what it's useful for. We occasionally get people trying to use the ABC in type annotations, so the current state causes confusion.
Perhaps we could deprecate ByteString, or explicitly document its limited use.
Perhaps we could deprecate ByteString, or explicitly document its limited use.
I vote for deprecation because the name is bad (implying
bytes | str) and it would just be a continuing point of confusion.Reacted by Alex Waygood, Jelle Zijlstra, Martin DeMello and Sebastian Rittau- changed the title
[-]Give collections.abc.ByteString some methods[/-][+]Deprecate collections.abc.ByteString[/+]on Apr 25, 2022 The name is good to me. It implies the bytes-like object with
strmethods (find(),lower(),isspace(), etc).The terms "buffer", "bytes-like", "bytestring" and "bytes string" are used loosely in the documentation, but there are several meanings of bytes-likeness:
- Supports the buffer protocol.
- Additionally supports
len()which returns the size in bytes. - Additionally supports indexing.
- Has most methods of
str(exceptencode()of course).
Unfortunately there are no strongly defined terms and corresponding abstract classes, protocols or types in the code to express the requirements precisely.
Hi! I'm one of the contributors to the pygame project.
I'm not sure whether this is the best best place to be asking, but it is relevant to the usage of
ByteString.So we have a function implemented with the python C API, and it uses
y#, which according to the docs is a format string for a generic "bytes-like" sized object. The term "bytes-like" is defined here in the glossary which gives me the impression that any function usingy#must accept a wide range of "byte-like" objects. Simply usingbytes | bytearraywould be narrow, and probably miss some kinds of objects. The same also mentions that "bytes-like object" is an object supporting the C level buffer protocol (which, is also not fully exposed on the python end in my understanding)I was looking for a suitable ABC to typehint this, and the closest thing I could find that already exists is
ByteString.
The next closest thing I found istyping.SupportsByteswhich by the naming, gives me the impression that this is what I'm looking for, but weirdly enough, the concretebytesobject itself does not confirm to this protocol (due to missing the__bytes__method)I suppose a set of ABCs for the buffer protocol (if this were to be added) would be the closest replacement to
ByteString, and would also work for my usecase (typing the C levely#arg format)The problem with buffers is that it doesn't have a visible Python API, so the ABC would be really weird, being entirely empty. See previous discussion at python/typing#593 and then PEP 688.
18 remaining items
- added a commit that references this issue
on May 9, 2023 - added a commit that references this issue
on May 12, 2023 - added a commit that references this issue
on May 12, 2023 - added a commit that references this issue
on May 13, 2023 - added 2 commits that reference this issue
on May 15, 2023 - changed the title
[-]Deprecate collections.abc.ByteString[/-][+]Deprecate collections.abc.ByteString and typing.ByteString[/+]on Jul 14, 2023 - changed the title
[-]Deprecate collections.abc.ByteString and typing.ByteString[/-][+]Deprecate and schedule removal of collections.abc.ByteString and typing.ByteString[/+]on Jul 14, 2023 - added a commit that references this issue
on Sep 18, 2023 - added a commit that references this issue
on Oct 1, 2023
The current docstring of
collections.abc.ByteStringis:Let's do that last thing. This will be useful for typing code that accepts both bytes and bytearray, especially with my proposal in PEP-688 to make bytes no longer acceptable as a shortcut for bytearray in the type system.
cc @rhettinger for
collections.abcLinked PRs
ByteStringdeprecation warnings #104294ByteString#104424