Repository navigation
Several C API functions do not return NUL-terminated buffers #8685
Description
Summary
The C API documents that the buffers returned by PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize() to contain an extra trailing NUL byte. RustPython instead returns pointers to the objects' exact-length Rust buffers. Reading the documented byte at data[len] is therefore a heap-buffer-overflow.
The relevant implementations return raw pointers without adding a terminator: PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize().
The bytearray case was raised during review of PR #8009, but dismissed as intentional. However, I'm not sure whether this being a soundness issue was raised at that time.
Reproducer
Save as capi_trailing_null.c:
#include <Python.h>
#include <stdio.h>
int main(void)
{
PyObject *bytes, *bytearray, *unicode;
char *bytes_data, *bytearray_data;
const char *unicode_data;
Py_ssize_t bytes_len, bytearray_len, unicode_len;
unsigned char bytes_trailing, bytearray_trailing, unicode_trailing;
Py_Initialize();
bytes = PyBytes_FromStringAndSize("abc", 3);
bytearray = PyByteArray_FromStringAndSize("abc", 3);
unicode = PyUnicode_FromStringAndSize("abc", 3);
if (!bytes || !bytearray || !unicode)
return 2;
bytes_data = PyBytes_AsString(bytes);
bytearray_data = PyByteArray_AsString(bytearray);
unicode_data = PyUnicode_AsUTF8AndSize(unicode, &unicode_len);
bytes_len = PyBytes_Size(bytes);
bytearray_len = PyByteArray_Size(bytearray);
if (!bytes_data || !bytearray_data || !unicode_data ||
bytes_len < 0 || bytearray_len < 0)
return 2;
bytes_trailing = (unsigned char)bytes_data[bytes_len];
bytearray_trailing = (unsigned char)bytearray_data[bytearray_len];
unicode_trailing = (unsigned char)unicode_data[unicode_len];
printf("bytes: data[len] = 0x%02x\n", bytes_trailing);
printf("bytearray: data[len] = 0x%02x\n", bytearray_trailing);
printf("unicode UTF-8: data[len] = 0x%02x\n", unicode_trailing);
Py_Finalize();
return bytes_trailing || bytearray_trailing || unicode_trailing;
}Expected
With CPython:
$ cc -fsanitize=address -fno-omit-frame-pointer $(python3-config --cflags) capi_trailing_null.c $(python3-config --embed --ldflags) -o trailing-null-cpython
$ ASAN_OPTIONS=detect_leaks=0 ./trailing-null-cpython
bytes: data[len] = 0x00
bytearray: data[len] = 0x00
unicode UTF-8: data[len] = 0x00Actual
From a RustPython checkout on macOS (use .so instead of .dylib on Linux):
$ cargo build -p rustpython-capi
$ cc -fsanitize=address -fno-omit-frame-pointer $(python3-config --includes) capi_trailing_null.c target/debug/librustpython_capi.dylib -o trailing-null-rustpython
$ ASAN_OPTIONS=detect_leaks=0 ./trailing-null-rustpython
==...==ERROR: AddressSanitizer: heap-buffer-overflow
...
0x... is located 0 bytes after 3-byte region [0x...,0x...)Python Documentation
CPython documents the trailing byte for PyBytes_AsString(), PyByteArray_AsString(), and PyUnicode_AsUTF8AndSize().
Activity
ngoldbaum commentedon Sep 10, 2026
Also you might be interested in python/cpython#157236. Thinking about that PR is how I ran into this.
bschoenmaeckers commentedon Sep 13, 2026
Thanks @ngoldbaum, this is indeed a bug. I will pick this up in the near future.
bschoenmaeckers commentedon Sep 15, 2026
If I understand python/cpython#157236 correctly, this will probably be documented as an implementation detail of CPython. So we might not be required to allocate a extra null byte here.
ngoldbaum commentedon Sep 15, 2026
The C API functions returning the extra NULL byte isn't an implementation detail though. That's been documented for a long time.
vstinner commentedon Sep 15, 2026
Well, Petr Viktorin has a strong opinion on what should qualify as an "implementation detail". His point is that it's possible to have a byte object without extra trailing null byte, but allocate a copy with a trailing null byte when
PyBytes_AsString()is called.PyBytes_AsString() requires to have this trailing null byte:
Well, for me it would be more efficient to create directly the bytes object with the trailing null copy. Having to create a copy for PyBytes_AsString() does double the memory usage and make memory management more complicated.
CPython always allocates the extra null byte when it creates bytes and bytearray objects.
PyByteArray_AsString() also says:
bschoenmaeckers commentedon Sep 15, 2026
Thanks for the explanation! My understanding was wrong 😅
youknowone commentedon Sep 16, 2026
Thank you so much everyone. We tried not to add trailing nul, but regarding c api, adding one will be more easy for everything. That will also help to resolve remaining ctypes imcompatibility.