Possible unnecessary OverflowError in random.getrandbits

Documentation

Problem

Actual upper bound of a random.Random.randbytes is 2**28 - 1

Following docs and exception message I would expect the limit of 2 ** 32 - 1 as for C 4-bytes integer

    ## -------------------- bytes methods ---------------------

    def randbytes(self, n):
        """Generate n random bytes."""
        return self.getrandbits(n * 8).to_bytes(n, 'little')

>>> bytes_string = random.randbytes(2 ** 28)
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "C:\Program Files\Python39\lib\random.py", line 286, in randbytes
    return self.getrandbits(n * 8).to_bytes(n, 'little')
OverflowError: Python int too large to convert to C int
>>> bytes_string = random.randbytes(2 ** 28 - 1)
>>> len(bytes_string)
268435455

Reproduced on
Python 3.9.4 (tags/v3.9.4:1f2e308, Apr 6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
Python 3.11.2 (main, Jul 19 2024, 12:24:02) [GCC 12.2.0] on linux

Will you please point me - if there is a bug in docs/implementation, or this is my misunderstanding only.
Thank you in advance for you patience.

Linked PRs

@serhiy-storchaka

@serhiy-storchaka The overflow occurs in the upstream call to random.getrandbits(8 * 2 ** 28 - 1).

The argument clinic spec reads:

_random.Random.getrandbits

  self: self(type="RandomObject *")
  k: int
  /

Should k be switched to object and converted with PyLong_AsLongLongAndOverflow? The 32-bit signed int limitation on Windows seems unnecessarily restrictive.

Why not have int32 converter in the AC? Now we have PyLong_AsInt32/AsUInt32/etc.

@skirpichev

@skirpichev We want something like ssize_t so that we can ask for arrays as big as we can allocate. The limit should be driven by memory constraints rather than function argument constraints. 32-bit signed is insufficient.

Why not have int32 converter in the AC?

It was in my plans.

For this case it is better to use the size_t converter (automatic ValueError for negative value), or even the uint64 converter (to be able creating bytes objects up to 2 GiB on 32-bit platforms).

pythongh-133489: Remove size restrictions on getrandbits() and randby…

@Raymond

#133658 makes getrandbits() using 64-bit bits count. Since in CPython a hard limit for bits count in integers is 64-bit, this is the right type.

What should we do with other versions, @Raymond? I think that it is not late to backport it to 3.14, and the backport is strightforward, because the uint64 converter exists in 3.14. In 3.13 we could use unsigned_long_long, which is practically the same. Or we can document the limitation if it is too late for backport.

raising OverflowError feels like a bug so back porting all the way to 3.13 as a bug fix makes sense to me.

gh-133489: Remove size restrictions on getrandbits() and randbytes() (G…

pythongh-133489: Remove size restrictions on getrandbits() and randby…

[3.13] pythongh-133489: Remove size restrictions on getrandbits() and…

[3.14] gh-133489: Remove size restrictions on getrandbits() and randb…

[3.13] gh-133489: Remove size restrictions on getrandbits() and randb…

There is an (unintended?) api break. In python 3.13.3, getrandbits() accepts an "integer-like" object as in:

Python 3.13.3 (main, Apr 11 2025, 13:41:26) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import random ; r = random.Random()
>>> from numpy import int64
>>> r.getrandbits(int64(20))
582096

but in python 3.13.4, getrandbits() fails as in:

Python 3.13.4 (main, Jun  8 2025, 19:43:01) [GCC 14.2.1 20250405] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import random ; r = random.Random()
>>> from numpy import int64
>>> r.getrandbits(int64(20))
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    r.getrandbits(int64(20))
    ~~~~~~~~~~~~~^^^^^^^^^^^
TypeError: an integer is required

This causes trouble for sagemath, where integers are by default a special type ZZ.

Can you please open a new issue to track that?

Ah, that's because _PyLong_UnsignedLongLong_Converter() has no fallback to index-like objects.

In 3.14+ we are using PyLong_AsNativeBytes for uint64 converter and it's not an issue. Maybe just add a uint64 converter for 3.13 (it has PyLong_AsNativeBytes) or this is too much?

pythongh-135326: Revert "[3.13] pythongh-133489: Remove size restrict…

pythongh-133489: Remove size restrictions on getrandbits() and randby…

Navigation Menu

Repository navigation

Possible unnecessary OverflowError in random.getrandbits #133489

Documentation

Problem

Linked PRs

2 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Possible unnecessary OverflowError in random.getrandbits #133489

Description

Documentation

Problem

Linked PRs

Activity

rhettinger commented on May 6, 2025

skirpichev commented on May 7, 2025

rhettinger commented on May 7, 2025

serhiy-storchaka commented on May 7, 2025

2 remaining items

serhiy-storchaka commented on May 8, 2025

gpshead commented on May 31, 2025

tornaria commented on Jun 9, 2025

gpshead commented on Jun 9, 2025

skirpichev commented on Jun 9, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions