ctypes: bit field data does not survive round trip

@mdickinson

Bit-fields in structures don't seem to give you back the data you put in?

from ctypes import Structure, c_uint, c_ulonglong, c_ushort


class Foo(Structure):
    _fields_ = [("A", c_uint, 1), ("B", c_ushort, 16)]


class Bar(Structure):
    _fields_ = [("A", c_ulonglong, 1), ("B", c_uint, 32)]


if __name__ == "__main__":
    for a in [Foo(), Bar()]:
        a.A = 0
        a.B = 1
        print(a.A, a.B)

The above should print

0 1
0 1

But it actually prints

$ python3.10 mini.py 
0 0
0 0

For comparison and to test my understanding, I expect the following C code to be equivalent to the Python code above:

#include<stdio.h>

struct Foo {
  unsigned int A: 1;
  unsigned short B: 16;
};

struct Bar {
  unsigned long long int A: 1;
  unsigned int B: 32;
};

int main(int argc, char** argv) {
    struct Foo foo;
    foo.A = 0;
    foo.B = 1;
    printf("%d %d\n", foo.A, foo.B);

    struct Bar bar;
    bar.A = 0;
    bar.B = 1;
    printf("%d %d\n", bar.A, bar.B);
    return 0;
}

The C version prints what we expect:

$ gcc -fsanitize=undefined test.c && ./a.out
0 1
0 1

Your environment

I am on ArchLinux with Python 3.10.7. Python 3.11 and main are also affected. I also randomly tried Python 3.6 with the same result. (Python 3.6 is the oldest one that was easy to install.)

More comprehensive test case

Here's how I actually found the problem reported above. Using Hypothesis:

import ctypes
import string

from hypothesis import assume, example, given, note
from hypothesis import strategies as st

unsigned = [(ctypes.c_ushort, 16), (ctypes.c_uint, 32), (ctypes.c_ulonglong, 64)]
signed = [(ctypes.c_short, 16), (ctypes.c_int, 32), (ctypes.c_longlong, 64)]
types = unsigned + signed

unsigned_types = list(zip(*unsigned))[0]
signed_types = list(zip(*signed))[0]

names = st.lists(st.text(alphabet=string.ascii_letters, min_size=1), unique=True)


@st.composite
def fields_and_set(draw):
    names_ = draw(names)
    ops = []
    results = []
    for name in names_:
        t, l = draw(st.sampled_from(types))
        res = (name, t, draw(st.integers(min_value=1, max_value=l)))
        results.append(res)
        values = draw(st.lists(st.integers()))
        for value in values:
            ops.append((res, value))
    ops = draw(st.permutations(ops))
    return results, ops


def fit_in_bits(value, type_, size):
    expect = value % (2**size)
    if type_ not in unsigned_types:
        if expect >= 2 ** (size - 1):
            expect -= 2**size
    return expect


@given(fops=fields_and_set())
def test(fops):
    (fields, ops) = fops

    class BITS(ctypes.Structure):
        _fields_ = fields

    b = BITS()
    for (name, type_, size), value in ops:

        expect = fit_in_bits(value, type_, size)
        setattr(b, name, value)
        j = getattr(b, name)
        assert expect == j, f"{expect} != {j}"


if __name__ == "__main__":
    test()

Thanks to @mdickinson for pointing me in this direction.

Linked PRs

pythongh-97588: Some tests to demonstrate the issue

Since I'm looking into bit-fields and ctypes anyway at the moment, I'll plan to work on a fix. But if someone else wants to give it a go, I'm happy to let them.

As far as I can tell the problem might be with PyCField_FromDesc in cfield.c. The code looks both a bit complicated and is not expected to be a hot inner loop (it's only for setting up a Structure from a description.) So perhaps we might want to move most of its logic into Python.

Let's me dig some more.

pythongh-97588: Fix ctypes structs

@matthiasgoergens

@matthiasgoergens I think this is essentially the same issue as existing issues like #95496, #84039, #73939, #59324, etc; it would be good to try to consolidate some of these so we don't end up with an explosion of issues.

@mdickinson

@mdickinson Good idea. That's actually what I am already doing with the fix I have in the works, but consolidating the issues makes sense, too.

@mdickinson

@mdickinson I mentioned the issues that my PR fixes on the PR itself. Is there a way (or necessity) to also consolidate the issues themselves here?

Backport 'pythongh-97588: Fix ctypes structs' to 3.11

Squash 'pythongh-97588: Fix ctypes structs' for backport to 3.10

@matthiasgoergens

@matthiasgoergens Could you review pull request !103052

@thesamprice

@thesamprice I'll have a look, but keep in mind that I don't have any authority here.

More interested if the one line change passes the tests you wrote .

Oh, ok. Sure, I can run that against my tests.

I guess I should publish my Python Hypothesis tests, too; and not just the tests in my PR. So you can run them yourself.

Give me a bit of time!

Edit: I'll answer over on #103052

@thesamprice

@thesamprice I added your tests from #103052 to this PR, too. I hope you don't mind.

gh-97588: Align ctypes struct layout to GCC/MSVC (GH-97702)

pythongh-97588: Align ctypes struct layout to GCC/MSVC (pythonGH-97702)

gh-97588: Move ctypes struct/union layout logic to Python (GH-123352)

Thank you for the report, and the initial fix!
The layout calculation is now done in Python, and a test suite is in place, so we're in a better place to solve the other struct/union layout issues. I'll close this one.

pythongh-97588: remove unused functions in _ctypes/cfield.c

gh-97588: remove unused functions in _ctypes/cfield.c (GH-124010)

pythongh-97588: remove unused functions in _ctypes/cfield.c (pythonGH…

Navigation Menu

Repository navigation

ctypes: bit field data does not survive round trip #97588

Your environment

More comprehensive test case

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ctypes: bit field data does not survive round trip #97588

Description

Your environment

More comprehensive test case

Linked PRs

Activity

matthiasgoergens commented on Sep 27, 2022

matthiasgoergens commented on Sep 27, 2022

mdickinson commented on Oct 8, 2022

matthiasgoergens commented on Oct 8, 2022

matthiasgoergens commented on Oct 9, 2022

thesamprice commented on Mar 27, 2023

matthiasgoergens commented on Mar 27, 2023

thesamprice commented on Mar 27, 2023

matthiasgoergens commented on Mar 27, 2023

matthiasgoergens commented on Mar 27, 2023

encukou commented on Sep 9, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions