Repository navigation
Silent ctypes.struct layout change from native to Windows when bitfields are used #131747
Description
Bug report
Bug description:
Python 3.14 implements a special field _layout_ in ctypes.Structure that allows the programmer to explicitly choose the layout rules for bitfields used by either GCC/Clang or MSVC. The default layout is appropiate for the OS platform (MSVC on Windows, GCC otherwise) which is expected behaviour. It is also possible to use field _pack_ to change the layout to match the compiler. This is currently only implemented for the MSVC layout, not for GCC.
The problem is that using _pack_ on non-Windows systems silently switches the layout rules from GCC to MSVC, which is an unexpected layout by default on non-Windows systems. I propose that this case be handled as an error (NotImplementedError) rather than the silent layout switch. The error message may contain a hint that if the MSVC layout is indeed what is intended, one need to select it explicitly by setting field _layout_.
From the Zen of Python:
Explicit is better than implicit.
Readability counts.
Special cases aren't special enough to break the rules.
Errors should never pass silently.
Tested on 3.14a6
CPython versions tested on:
3.14
Operating systems tested on:
macOS
Activity
ZeroIntensity commentedon Mar 25, 2025
cc @encukou (you messed with bitfields recently, right?)
If this is valid, we should probably mark this as a blocker.
BCSharp commentedon Mar 26, 2025
@ZeroIntensity I'm glad that you see it as a bug and not a feature request. I discovered it while implementing the same functionality in IronPython, and just couldn't bring myself to follow CPython in this case (normally IronPython follows what CPython does, if possible). It would be great if CPython changes this so that there are no uneecessary differences between IronPython and CPython.
ZeroIntensity commentedon Mar 26, 2025
Yeah, I doubt it's intended behavior. Assuming that's correct (I wasn't part of designing or reviewing
_layout_/_pack_, so I'm going to yield to someone else), it would be good to get this fixed before the beta freeze locks it in.encukou commentedon Mar 26, 2025
Hi! I have limited access to the computer & I'm writing from memory. Please verify my claims :)
Yes, 3.14 has an overhaul of struct layout; look at 3.13 for the previous status quo.
As far as I remember,
_pack_has always switched to the MSVC layout; changing that would be a breaking change too.In both 3.13 and the current 3.14, the meaning of
_pack_is “MSVC#pragma pack”.GCC/clangs's
__attribute__((packed))is not implemented/supported at all.We could make that more explicit in the docs. (Unfortunately, after such time, people depend on the actual behaviour; it doesn't matter that there could be a better design that would match the docs.)
Would such a docs change (“
_pack_is “MSVC #pragma pack”) allow you to bring yourself to follow CPython? (If so, could you send a PR?)We could deprecate
_pack_without_layout_on non-Windows, but I fear that it would break existing users unnecssarily.Of course, if
__attribute__((packed))is implemented soon (which is not on anyone's roadmap AFAIK), it might be nice to switch the default for “Linux with_pack_” to it.But my current vague long-term plan is to make
_layout_a callable that constructs the field list, and gradually move away from configuration strings to something like_layout_ = gcc_sysv(packed=1). And if_pack_will eventually be (soft-)deprecated, then I want to avoid “churn” for existing users: let them switch at once to a well-designed replacement, whenever it's ready.BCSharp commentedon Mar 26, 2025
To clarify (or add to the confusion)
#pragma packis also supported by GCC/Clang and is something different than__attribute__((packed)). I understand that_pack_deals only with the former and not the latter, and that GCC packing is not yet supported in any form. It also makes sense that eventually a new scheme will be needed to support__attribute__((packed)), since it can be applied per field. For backward compatibility I see_pack_being supported as an alternative to the new scheme, but then it will be even more confusing if it always means MSVC#pragma packand never GCC#pragma pack.If the silent layout change was always in Python, I see it as it was always a bug that went unreported. This is not surprising since till 3.14 there far more serious problems with handling bitfields (overlapping fields, spurious padding bits inserted that did not match the compiler) so I doubt this is widely used. Even if making it an error "breaks" some existing code that just happens to work correctly, there is a trivial remedy by adding
_layout_ = 'msvc'(or whatever the codestring for MSVC will be in the final release).At the very least, if not an error, please make it a warning that the layout was changed and that in the future this will become GCC pragma pack, so the user better be explicit.
pythongh-131747: ctypes: Warn when _pack_ implicitly changes default …
pythongh-131747: ctypes: Warn when _pack_ implicitly changes default …
pythongh-131747: ctypes: Warn when _pack_ implicitly changes default …
encukou commentedon May 1, 2025
Not 100% convinced that breaking the old behaviour is OK, but, we can revert a deprecation warning.
Does #133205 look OK?
BCSharp commentedon May 1, 2025
Looks good to me! Thanks!
gh-131747: ctypes: Deprecate _pack_ implicitly setting _layout_ = 'ms' (
encukou commentedon May 17, 2025
See #133800 for more discussion.
pythongh-131747: ctypes: Deprecate _pack_ implicitly setting _layout_…